351 Matching Annotations
  1. Jan 2025
    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      Now let's look at another example.

      This looks more complex, but we're going to use the same process to identify the correct answer.

      So this is a multi-select question and we're informed that we need to pick two answers, but we're still going to follow the same process.

      The first step is to check if we can eliminate any of the answers immediately.

      Do any of the answers not make sense without reading the question?

      Well, nothing immediately jumps out as wrong, but answer E does look strange to me.

      It feels like it's not a viable solution.

      I can see the word encryption mentioned and it's rare that I see lambda and encryption mentioned in the same statement.

      So at this stage, let's just say that answer E is in doubt.

      So it's the least preferred answer at this point.

      So keep that in your mind.

      It's fine to have answers which you think are not valid.

      We don't know enough to immediately exclude it, but we can definitely say that we think there's something wrong with it.

      Given that we need to select two answers out of the five, we don't need to worry about E as long as there are two potentially correct answers.

      So let's move on.

      Now, the real step one is to identify what matters in the question text.

      So let's look at that.

      Now, the question is actually pretty simple.

      It gives you two requirements.

      The first is that all data in the cloud needs to be encrypted at rest.

      And the second is that any encryption keys are stored on premises.

      For any answers to be correct, they need to meet both of these requirements.

      So let's follow a similar process on the answer text first looking for any word fluff and then looking for keywords which can help identify either the correct answers or more answers that we can exclude.

      So the first three answers, they all state server side encryption, but the remaining two answers don't.

      And so the first thing that I'm going to try to do with this question is to analyze whether server side encryption means anything.

      Does it exclude the answers or does it point to those answers being correct?

      Well, server side encryption means that S3 performs the encryption and decryption operations.

      But depending on the type of server side encryption, it means that S3 either handles the keys or the customer handles the keys.

      But at this stage, using server side encryption doesn't mean that the answers are right or wrong.

      You can use it or you can't use it.

      That doesn't immediately point to correct versus incorrect.

      What we need to do is to look at the important keywords.

      Now, if we assume that we are excluding answer E for now unless we need it, then we have four different possible answers, each of which is using a different type of encryption.

      So I've highlighted these.

      So we've got S3 managed keys, SSE-S3, KMS managed keys, which is SSE-KMS, customer provided keys, which is SSE-C, and then using client side encryption.

      Now, the first requirement of the question states encryption at rest and all of the answers A, B, C and D, they all provide encryption at rest.

      But it also states that encryption keys are to be stored on premises.

      Answers A and B use server side encryption where AWS handle the encryption process and the encryption keys.

      So SSE-S3 and SSE-KMS both mean that AWS are handling the encryption keys.

      And because of this, the keys are not stored on premises and so they don't meet the second criteria in the question.

      And this means that they're both invalid and can be excluded.

      Now, this leaves answers C, D and E, and we already know that we're assuming that we're ignoring answer E for now and only using it if we have to.

      So we just have to evaluate if C and D are valid.

      And if they are, then those are the answers that we select.

      So SSE-C means that the encryption is performed by S3 with the keys that the customer provides.

      So that works.

      It's a valid answer.

      So that means at the very least C is correct.

      It can be used based on the criteria presented by the question.

      Now, answer D suggests client side encryption, which means encrypting the data on the client side and just passing the encrypted data to S3.

      So that also works.

      So answers C and D are both potentially correct answers.

      So both answers C and D do meet the requirements of the question.

      And because of this, we don't have to evaluate answer E at all.

      It's always been a questionable answer.

      And since the question only requires us to specify two correct answers, we can go ahead and exclude E.

      And that gives us the correct answers of C and D.

      So that's another question answered.

      And it's followed the same process that we used on the previous example.

      So really answering questions within AWS is simply following the same process.

      Try and eliminate any crazy answers.

      So any answers that you can eliminate based just on the text of those answers, then exclude them right away because it reduces the cognitive overhead of having to pick between potentially four correct answers.

      If you can eliminate the answers down to three or two, you significantly reduce the complexity of the question.

      The next step is to find what really matters in the question, find the keywords in both the preamble and the question.

      Then highlight and remove any question fluff.

      So anything in the question which doesn't matter, eliminate any of the words which aren't relevant technically to the product or products that you select.

      So this is something that comes with experience, being able to highlight what matters and what doesn't matter in questions.

      And the more practice that you do, the easier this becomes.

      Next, identify what really matters in the answers.

      So again, this comes down to identifying any shared common words and removing those and then identifying any of the keywords that occur in the answers.

      And then once you've got the keywords in the answers and the keywords in the questions, then you can eliminate any bad answers that occur in the question.

      Now, ideally at this point, what remains are correct answers.

      You might start off with four or five answers.

      You might eliminate two or three.

      The question asks for two correct answers.

      And that's it.

      You've finished the question.

      But if you have more answers than you need to provide, then you need to quickly select between what remains and you can do that by doing this keyword matching.

      So look for things which stand out.

      Look for things which aren't best practice according to AWS.

      Look for things which breach a timescale requirement in the question.

      Look for things which can't perform at the levels that the question requires or that cost too much based on the criteria and the question.

      Essentially, what you're doing is looking for that one thing that will let you eliminate any other answers and leave you with the answers that the question requires.

      Generally, when I'm answering most questions, it's a mixture between the correct answer jumping out at me and eliminating incorrect answers until I'm left with the correct answers.

      You can approach questions in two different ways.

      Either looking for the correct answers or eliminating the incorrect ones.

      Do whichever works the best for you and follow the same process throughout every question in the exam.

      The one big piece of advice that I can give is don't panic.

      Everybody thinks they're running out of time.

      Most people do run out of time.

      So follow the exam technique process that I detailed in the previous lesson to try and get you additional time, leave the really difficult questions until the end, and then just follow this logical process step by step through the exam.

      Keep an eye on the remaining amount of time that you have at every point through the exam and I know that you will do well.

      Most people fail the exam because of their exam technique, not their lack of technical capability.

      With that being said, though, that's everything I wanted to cover in this set of lessons.

      Good luck with the practice tests.

      Good luck with the final exam.

      And if you do follow this process, I know that you'll do really well.

      With that being said, though, go ahead and complete this video and then when you're ready, I'll look forward to joining you in the next.

    1. Welcome back and from the very start this course has been about more than just the technical side.

      So this is part one of a two-part lesson set and in this lesson I'll focus on some exam technique hints and tips that you might find useful in the exam.

      Now in terms of the exam itself it's going to have questions of varying levels of difficulty and this is also based on your own strengths and weaknesses.

      Conceptually though understand that on average the AWS exams will generally feel like they have 25% easy questions, 50% medium questions and 25% really difficult questions.

      Assuming that you've prepared well and have no major skill gaps this is the norm.

      For most people this is how it feels.

      The problem is the order of the difficulty is going to feel random so you could have all of your easy ones at the start or at the end or scattered between all of the other questions and this is part of the technique of the AWS exams how to handle question difficulty in the most efficient way possible.

      Now I recommend conceptually that my students think of exams in three phases.

      You want to spend most of your time on phase two.

      So structurally in phase one I normally try to go through all of the 65 questions and identify ones that I can immediately answer.

      You can use the exam tools including mark for review and just step through all of the questions on the exam answering anything that's immediately obvious.

      If you can answer a question within 10 seconds or have a good idea of what the answer will be and just need to consider it for a couple more seconds this is what I term a phase one question.

      Now the reason that I do these phase one questions first is that they're easy they take very little time and because you know the subject so well you have a very low chance of making a mistake.

      So once you've finished all of these easy questions the phase one questions what you're left with is the medium or yellow questions and the hard or red questions.

      My aim is that I want to leave the hard questions until the very end of the test.

      They're going to be tough to answer anyway and so what I want to do at this stage in phase two is to go through whatever questions remain so whatever isn't easy and I'm looking to identify any red questions and mark them for review and then just skip past them.

      I don't want to worry about any red questions in phase two.

      What phase two is about is powering through the medium questions.

      These will require some thought but they don't scare you they're not impossible.

      The medium questions so the yellow questions should make up the bulk of the time inside the exam.

      They should be your focus because these are the questions which will allow you to pass or fail the exam.

      For most people the medium questions represent the bulk of the exam questions.

      Generally your perception will be that most of the questions will be medium.

      There'll be some easy and some hard so you need to focus in phase two which represents the bulk of the exam on just these medium questions.

      So my suggestion generally is in phase two you've marked the hard questions for review and just skipped past them and then you focused on the medium questions.

      Now after you've completed these medium questions you need to look at your remaining time and it might be that you have 40 minutes left or you might only have four minutes or even less.

      In the remaining time that you have left you should be focusing on the remaining red questions the difficult questions.

      If you have 40 minutes left then you can take your time.

      If you have four minutes you might have to guess or even just click answers at random.

      Now both of these approaches are fine because at this point you've covered the majority of the questions.

      You've answered all of the easy questions and you've completed all of the medium questions.

      What remains are questions that you might get wrong regardless but because you've pushed them all the way through to the end of your time allocation whether you're considering them carefully and answering them because you have 40 minutes left or whether you're just answering them at random they won't impact your process in answering the earlier questions.

      So if you don't follow this approach what tends to happen is you're focusing really heavily on the hard questions at the start of the exam and that means that you run out of time towards the end but if you follow this three-stage process by this point all that you have left is a certain number of minutes and a certain set of really difficult questions and you can take your time safe in the knowledge that you've already hopefully passed to the exam based on the easy and medium questions and the hard ones as simply a bonus.

      Now at a high level this process is designed to get you to answer all of the questions that you're capable of answering as quickly as possible and leave anything that causes you to doubt yourself or that you struggle with to the end.

      So pick off the easy questions, focus on the medium and then finish up with the really hard questions at the end.

      I know that it sounds simple but unless you focus really hard on this process or one like it then your actual exam experience could be fairly chaotic.

      If you're unlucky enough to get hard questions at the start and you don't use a process like this it can really spoil your flow.

      So before we finish this lesson just some final hints and tips that I've got based on my own experiences.

      First if this is your first exam assume that you're going to run out of time.

      Most people enter the exam not having an understanding of the structure and most people myself included with my first exam will run out of time.

      The way that you don't run out of time and the way that you succeed is to be efficient, have a process.

      Now assuming that you have the default amount of time you need to be aware that you have two minutes to read the question, read the answers and to make a decision.

      So this sounds like a lot but it's not a lot of time to do all of those individual components.

      You shouldn't be guessing on any answers until the end.

      If you're guessing on a question then it should be in the hard question category and you should be tackling this at the end.

      I don't want you guessing on any easy questions or any medium questions.

      If you're guessing then you shouldn't be looking at it until right at the very end.

      Another way of looking at this is if you are unsure about a question or you're forced to guess early on you need to be aware that a question that's later on so further on in the exam might prompt you to remember the correct answer for an earlier question.

      So if you do have to guess on any questions then use the mark for review feature.

      You can mark any question that you want for review as you go through the course and then at any point or right at the end you can see all the questions which are flagged for review and revisit them.

      So use that feature it can be used if you're doubtful on any of the answers or you want to prompt yourself as with the hard questions to revisit them toward the end of the exam.

      Now this should be logical but take all the practice tests that you can.

      One of my favorite test vendors in the space is the team over at TutorialsDojo.com.

      They offer a full range of practice questions for all of the major AWS exams so definitely give their site a look.

      One of the benefits of the exam questions created over at TutorialsDojo is that they are more difficult than the real exam questions so they can prepare you for a much higher level of difficulty and by the time you get into the exam you should find it relatively okay.

      So my usual method is to suggest that people take a course and then once they've finished the course take the practice test in the course, follow that up with the tutorials Dojo practice tests and for any questions they get wrong it can identify areas that they need additional study.

      So rinse and repeat that process, perform that additional study, redo the practice tests and when you're regularly scoring above 90% on those practice tests then you're ready to do the real exam.

      And at this point there are all of my suggestions for exam technique.

      In the next lesson I want to focus on questions themselves because it's the questions and your efficiency during the process of answering questions which can mean the difference between success and failure.

      So go ahead complete this video and in the next video when you're ready we'll look at some techniques on how you can really excel when tackling exam questions.

    1. Welcome back and in this lesson I want to talk about a few related features of CloudFormation and those are weight conditions, creation policies and the CFN signal tool.

      So let's jump in and get started straight away.

      Before we look at all of those features as a refresher I want to step through what actually happens with the traditional CloudFormation provisioning process and let's assume that we're building an EC2 instance and we're using some user data to bootstrap WordPress.

      Well if we do this the process starts with logical resources within the template and the template is used to create a Cloud Formation stack.

      Now you know by now that it's the job of the stack to take the logical resources in a template and then create, update or delete physical resources to match them within an AWS account.

      So in this case it creates an EC2 instance within an AWS account.

      From CloudFormation's perspective in this example it initiates the creation of an EC2 instance so when EC2 reports back that the physical resource has completed provisioning the logical resource changes to create complete and that means everything's good right?

      Well the truth is we just don't know.

      With simple provisioning when the relevant system EC2 in this case tells CloudFormation that it's finished then CloudFormation has no further access to any other information beyond the fact that EC2 is telling it that that resource has completed its provisioning process.

      With more complex resource provisions like this one where bootstrapping goes on beyond when the instance itself is ready then the completion state isn't really available until after the bootstrapping finishes and even then there's no built-in link to communicate back to CloudFormation whether that bootstrapping process was successful or whether it failed.

      An EC2 instance will be in a create complete state long before the bootstrapping finishes and so even when it's finished if it fails the resource itself still shows create complete.

      Creation policies, weight conditions and CFN signal provide a few ways that we can get around this default limitation and allow systems to provide more detailed signals on completion or not to CloudFormation.

      So let's have a look at how this works.

      The way that this enhanced signaling is done is via the CFN signal command which is included in the AWS CFN bootstrap package.

      The principle is simple enough you configure CloudFormation to hold or pause a resource and I'll talk more about the ways that this is done next but you configure CloudFormation to wait for a certain number of success signals.

      You want to make it so that resources such as EC2 instances tell CloudFormation that they're okay.

      So in addition to configuring it to wait for a certain number of success signals you also configure a timeout.

      This is a value in hours, minutes and seconds within which those signals can be received.

      Now the maximum permitted value for this is 12 hours and once configured it means that a logical resource such as an EC2 instance will just wait.

      It won't automatically move into a create complete state once the EC2 system says that it's ready.

      Instead if the number of success signals that you define is received by CloudFormation within the timeout period then the status of that resource changes into create complete and the stack process continues with the knowledge that the EC2 instance really is finished and ready to go because on the instance you've configured something to explicitly send that signal or signals to CloudFormation.

      CFN signal is a utility running on the instance itself actually sending a signal back to the CloudFormation service.

      Now if CFN signal communicates a failure signal suggesting that the bootstrapping process didn't complete successfully then the creation of the resource in the stack fails and the stack itself fails.

      So that's important to understand CFN signal can send success signals or failure signals and a failure signal explicitly fails the process.

      Now another possible outcome of this is the timeout period can be reached without the required number of success signals and in this situation CloudFormation views this as an implicit failure.

      The resource being created fails and then logically the stack fails the entire process that it's doing.

      Now the actual thing which is being signaled using CFN signal is a logical resource specifically a resource such as EC2 or auto scaling groups which is using a creation policy or a specific type of separate resource called a weight condition resource.

      Now AWS suggests that for provisioning EC2 and auto scaling groups you should use a creation policy because it's tied to that specific resource that you're handling but you might have other requirements to signal outside of a specific resource.

      For example if you're integrating CloudFormation with an external IT system of some kind in that case you might choose to use a weight condition and next I want to visually step through how both of these work because it will make a lot more sense when you see the architecture visually.

      Let's start with the example of an auto scaling group which uses a launch configuration to launch three EC2 instances.

      These are within a template and that's used to create a stack.

      Because I'm using a creation policy here a few things happen which are different to how CloudFormation normally functions.

      First the creation policy here adds a signal requirement and timeout to the stack.

      In this case the stack needs three signals and it has a timeout of 15 minutes to receive them.

      So the EC2 instances are provisioned but because of the creation policy the auto scaling group doesn't move into a create complete state as normal.

      It waits.

      It can't complete until the creation policy directive is fulfilled.

      The user data for the EC2 instances contains some bootstrapping and then this CFN signal statement at the bottom.

      So once the bootstrapping process whatever it is has been completed and let's say that it's installing the Categorum application well the CFN signal tool signals the resource in this case the auto scaling group that it's completed the build.

      So this CFN signal that's at the bottom left of your screen this is an actual utility which runs on the EC2 instance as part of the bootstrapping process.

      And this causes each instance to signal once and the auto scaling group resource in the stack requires three of these signals within 15 minutes.

      If it gets them all and assuming that they're all success signals then the stack moves into a create complete state.

      If anything else happens so maybe a timeout happens or maybe one of the three instances has a bug then it will signal a failure and in any of those cases the stack will move into a create failed state.

      Creation policies are generally used for EC2 instances or for auto scaling groups and if you do any of the advanced demo lessons in any of my courses you're going to see that I make use of this feature to ensure resources which are being provisioned are actually provisioned correctly before moving on to the next stage.

      Now there are situations when you need some additional functionality maybe you want to pass data back to cloud formation or want to put general wait states into your template which can't be passed until a signal is received and that's where wait conditions come in handy.

      Wait conditions operate in a similar way to creation policies.

      A wait condition is a specific logical resource not something defined in an existing resource.

      A wait condition can depend on other resources and other resources can also depend on a wait condition so it can be used as a more general progress gate within a template a point which can't be passed until those signals are received.

      A wait condition will not proceed to create complete until it gets its signals or the timeout configured on that wait condition expires.

      Now a wait condition relies on a wait handle and a wait handle is another logical resource whose sole job is to generate a pre-signed URL which can be used to send signals to.

      It's pre-signed so that whatever using it doesn't need to use any AWS credentials they're included in the pre-signed URL.

      So let's say that we have an EC2 instance or external server.

      These are responsible for performing a process maybe some final detailed configuration or maybe they assign licensing something which has to happen after a part of the template but before the other part.

      So these generate a JSON document which contains some amazing information or some amazing occurrence.

      This is just an example it can be as complex or as simple as needed.

      This document is passed back as the signal it has a status, a reason, a unique ID and some data.

      Now what's awesome about this is that not only does this signal allow resource creation to be paused and then continued when this event has occurred but the data which has passed back can also be accessed elsewhere in the template.

      We can use the get at function to query for the data attribute of the wait condition and get access to the details on the signal.

      Now this allows a small amount of data exchange and processing between whatever is signaling and the cloud formation stack.

      So you can inject specific data about a given event into the JSON document, send this back as a signal and then access this elsewhere in the cloud formation stack and this might be useful for certain things like licensing or to get additional status information about the event from the external system.

      And that's wait conditions.

      In many ways they're just like creation policies.

      They have the same concept.

      They allow a specific resource creation to be paused, not allowing progress until signaling is received.

      Only wait conditions they're actually a separate resource and can use some more advanced data flow features like I'm demonstrating here.

      AWS recommend creation policies for most situations because they're simpler to manage but as you create more complex templates you might well have need to use wait conditions as well and for the exams it's essential that you understand both creation policies and wait conditions which is why I wanted to go into detail on both.

      Now that's all of the theory that I wanted to cover about creation policies and wait conditions and these are both things that you're going to get plenty of practical experience of in various demo lessons in all of my courses but I wanted to cover the theory and the architecture so that you can understand them when you come across them in those demos.

      For now though thanks for watching go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

  2. Nov 2024
    1. Welcome back to stage 5 of this advanced demo series.

      And in this stage you're going to be adding a load balancer and auto scaling group to provision and terminate instances automatically based on the load of the system.

      By adding a load balancer you'll also abstract connections away from individual instances which will allow elastic scaling and self-healing if any of the instances have problems.

      Now the first step to moving towards this elastic architecture is to create the load balancer.

      To do that move to the EC2 console, scroll down and toward the bottom under load balancing click on load balancers.

      Go ahead and click on create load balancer and it's going to be an application load balancer that we're creating.

      So click on create.

      We're going to be calling the load balancer A4L WordPress ALB.

      It's going to be an internet facing load balancer which means the nodes of the load balancer will be allocated with public IP addressing.

      And we want the IP address type for this demonstration to be IP version 4.

      Okay so now we need to select the subnets that the load balancer nodes will be placed into.

      So first make sure that the animals for life VPC is selected so A4L VPC.

      And then check the box next to US East 1A, 1B and 1C.

      For US East 1A I want you to select the SN-PUB-A which is the public subnet inside Availability Zone A so US East 1A.

      For US East 1B I want you to select the public subnet in AZB so SN-PUB-B.

      And then lastly for US East 1C we'll be selecting the SN-PUB-C.

      So this configures the subnets that the load balancer nodes will be placed into because they're public subnets and because we have the scheme set to internet facing these nodes will be provided with public IP addressing.

      Next under security groups click on the cross to delete the default security group.

      And then click in the drop down and go ahead and select A4L VPC-SG load balancer.

      Now there will be some random afterwards that's okay just make sure you select A4L VPC-SG load balancer.

      Now scroll down and under listeners and routing make sure that the protocol is set to HTTP and the port is set to 80.

      Application load balancers work using target groups and so we need to define a target group to forward the traffic to.

      Now we don't currently have any target groups which have been created so we need to go ahead and click on create target group.

      Now under basic configuration the target type is going to be instances so make sure that that's selected.

      Under target group name just enter A4L WordPress ALBTG.

      Scroll down further still make sure the protocol is set to HTTP and port is set to 80 on this screen as well.

      Make sure the VPC is set to A4L VPC.

      The protocol version by default should be HTTP1 you can leave that as the default.

      Under health checks make sure the health check protocol is HTTP and the health check path is forward slash.

      Once that's set go ahead and click next.

      Now we won't be adding any instances to the target group these can either be added manually or a target group can be integrated with an autoscaling group and that's something that we'll be configuring later in this advanced demo.

      For now just scroll down to the bottom and click create target group.

      Then go back to the previous tab click on the refresh icon and then select the A4L WordPress ALBTG from the drop down.

      Now we won't be picking any add-on services so you don't need to check the AWS global accelerator.

      Just scroll down to the bottom and click create load balancer.

      Next click on view load balancer and then select the load balancer that you've just started creating and we'll need to create another parameter in the parameter store so we'll need the DNS name of the load balancer.

      So go ahead and click on the little symbol next to that to copy that into your clipboard.

      Next you'll need to move back to the parameter store.

      Now because we're automating this environment we need to provide a way so that all of the EC2 instances know the DNS name of the load balancer because this will be used as a workaround to the fact that the IP addresses are hard coded into the database so we need to provide an automatic way of exposing the load balancer DNS name to the EC2 instances.

      Click on create parameter for the parameter name forward slash A4L forward slash WordPress forward slash ALB for application load balancer and then DNS name so forward slash A4L forward slash WordPress forward slash ALB DNS name for description put DNS name of the application load balancer for WordPress.

      We're going to be picking a standard tier parameter.

      It's going to be a string parameter.

      It's going to be a text for data type and in value go ahead and paste the DNS name of the load balancer which you just copied into your clipboard scroll down to the bottom and click on create parameter.

      Now the next thing we're going to do is to update the launch template and this is quite a complex update so you need to understand exactly what we're doing.

      Currently and I've mentioned this a few times throughout this demo series the IP address of the first EC2 instance that's used for a WordPress deployment is hard coded into the database.

      Now this is fine if it's a static IP address but if it's not or if you're using multiple EC2 instances then you can't use IP addresses because they change both on an individual EC2 instance and if you're scaling using multiple instances.

      So we need to replace this hard coded value with the DNS name of the load balancer.

      So that's what we're going to do.

      We're going to update the launch template with some final configuration so that it can adjust this configuration replacing the IP address with the DNS name of the load balancer.

      So go back to the EC2 console, click on launch templates, select the WordPress launch template and click on actions modify template create new version.

      Under the template version description we're going to use app only, users EFS file system defined in /a4l/wordpress/efs/fsid and then ALB home added to the WP database.

      So we're going to make some on the fly adjustments to the WordPress database when every instance is provisioned to make sure that the load balancer DNS name is set to be the home URL for WordPress.

      So again scroll all the way down to the bottom because we're using an older template as the foundation for this one.

      All of the values will be pre-populated.

      Expand advanced details and scroll all the way down to user data and then just expand this text entry to make it slightly easier to interact with.

      As with the previous step position your cursor at the end of this top line and press enter twice.

      We need to add the first two lines of script which will bring in the application load balancer DNS name into an environment variable using systems manager parameter store.

      So now this instance when it's provisioning has the DNS name of the load balancer.

      Now next move all the way down to the bottom of this user data.

      So the last step that we want a machine to do when it's provisioning is to perform this update of the database.

      So there's a fairly large block of text which you need to copy from this stages text instructions.

      It's stage five and you need to paste this into the bottom of this file.

      So right at the bottom after these last two fine statements paste in this block.

      So this should start with the cat command on the top line of what you've just pasted in and then all the way down at the bottom.

      It should end with forward slash home forward slash EC2 hyphen user forward slash update underscore WP underscore IP dot SH.

      Essentially what this does is to bring in the WordPress configuration file to get the current authentication details for the database.

      So all these lines at the top are just designed to get the authentication information.

      So the DB name, the DB user and the DB password.

      This line runs a database script to get the old value for the IP address of the original IP address of the EC2 instance.

      So this is pulling in the original hard coded IP address.

      Then we're going to take the load balancer DNS name and we're going to run a series of SQL commands to update the database moving from that hard coded IP.

      To using the ALB DNS name.

      Now what this is actually doing is this line here is creating a script file and it's going to put into this script file everything until this EOF directive.

      So scrolling down this means that everything between these two lines is going to be stored in this script.

      Then we're going to make the script executable using CHmod 755.

      We're going to echo the path to this script into ETC RC.local which is run every time the instance is started up.

      And then finally we're going to run this script the once to update this information right here and now.

      So this new version of the launch template essentially changes what this hard coded IP address is every time to be the DNS name of the load balancer.

      It means if we ever change the DNS name of this load balancer this script will automatically correct this hard coded value.

      Now this is a thing specific to WordPress and there are many situations where you'll have applications which have certain nuances that you need to be aware of when creating elastic architectures.

      This is the one for WordPress.

      So now that we've made these changes go ahead and click on create template version to create that new version of this launch template.

      Click on launch template some for the final time we need to update the default version.

      So make sure this launch template is selected.

      Click on actions scroll down select set default version click in the drop down the current default version is version three we want to select version four so select that and then click set as default version.

      Now that means the launch template is updated and we can now provision instances in a fully elastic way.

      Okay so this is the end of part one of this lesson.

      It was getting a little bit on the long side and I wanted to give you the opportunity to take a small break maybe stretch your legs or make a coffee.

      Now part two will continue immediately from this point so go ahead complete this video and when you're ready I look forward to you joining me in part two.

    1. Welcome back to stage 5 of this advanced demo series.

      And in this stage you're going to be adding a load balancer and auto scaling group to provision and terminate instances automatically based on the load of the system.

      By adding a load balancer you'll also abstract connections away from individual instances which will allow elastic scaling and self-healing if any of the instances have problems.

      Now the first step to moving towards this elastic architecture is to create the load balancer.

      To do that move to the EC2 console, scroll down and toward the bottom under load balancing click on load balancers.

      Go ahead and click on create load balancer and it's going to be an application load balancer that we're creating.

      So click on create.

      We're going to be calling the load balancer A4L WordPress ALB.

      It's going to be an internet facing load balancer which means the nodes of the load balancer will be allocated with public IP addressing.

      And we want the IP address type for this demonstration to be IP version 4.

      Okay so now we need to select the subnets that the load balancer nodes will be placed into.

      So first make sure that the animals for life VPC is selected so A4L VPC.

      And then check the box next to US East 1A, 1B and 1C.

      For US East 1A I want you to select the SN-PUB-A which is the public subnet inside Availability Zone A so US East 1A.

      For US East 1B I want you to select the public subnet in AZB so SN-PUB-B.

      And then lastly for US East 1C we'll be selecting the SN-PUB-C.

      So this configures the subnets that the load balancer nodes will be placed into because they're public subnets and because we have the scheme set to internet facing these nodes will be provided with public IP addressing.

      Next under security groups click on the cross to delete the default security group.

      And then click in the drop down and go ahead and select A4L VPC-SG load balancer.

      Now there will be some random afterwards that's okay just make sure you select A4L VPC-SG load balancer.

      Now scroll down and under listeners and routing make sure that the protocol is set to HTTP and the port is set to 80.

      Application load balancers work using target groups and so we need to define a target group to forward the traffic to.

      Now we don't currently have any target groups which have been created so we need to go ahead and click on create target group.

      Now under basic configuration the target type is going to be instances so make sure that that's selected.

      Under target group name just enter A4L WordPress ALBTG.

      Scroll down further still make sure the protocol is set to HTTP and port is set to 80 on this screen as well.

      Make sure the VPC is set to A4L VPC.

      The protocol version by default should be HTTP1 you can leave that as the default.

      Under health checks make sure the health check protocol is HTTP and the health check path is forward slash.

      Once that's set go ahead and click next.

      Now we won't be adding any instances to the target group these can either be added manually or a target group can be integrated with an autoscaling group and that's something that we'll be configuring later in this advanced demo.

      For now just scroll down to the bottom and click create target group.

      Then go back to the previous tab click on the refresh icon and then select the A4L WordPress ALBTG from the drop down.

      Now we won't be picking any add-on services so you don't need to check the AWS global accelerator.

      Just scroll down to the bottom and click create load balancer.

      Next click on view load balancer and then select the load balancer that you've just started creating and we'll need to create another parameter in the parameter store so we'll need the DNS name of the load balancer.

      So go ahead and click on the little symbol next to that to copy that into your clipboard.

      Next you'll need to move back to the parameter store.

      Now because we're automating this environment we need to provide a way so that all of the EC2 instances know the DNS name of the load balancer because this will be used as a workaround to the fact that the IP addresses are hard coded into the database so we need to provide an automatic way of exposing the load balancer DNS name to the EC2 instances.

      Click on create parameter for the parameter name forward slash A4L forward slash WordPress forward slash ALB for application load balancer and then DNS name so forward slash A4L forward slash WordPress forward slash ALB DNS name for description put DNS name of the application load balancer for WordPress.

      We're going to be picking a standard tier parameter.

      It's going to be a string parameter.

      It's going to be a text for data type and in value go ahead and paste the DNS name of the load balancer which you just copied into your clipboard scroll down to the bottom and click on create parameter.

      Now the next thing we're going to do is to update the launch template and this is quite a complex update so you need to understand exactly what we're doing.

      Currently and I've mentioned this a few times throughout this demo series the IP address of the first EC2 instance that's used for a WordPress deployment is hard coded into the database.

      Now this is fine if it's a static IP address but if it's not or if you're using multiple EC2 instances then you can't use IP addresses because they change both on an individual EC2 instance and if you're scaling using multiple instances.

      So we need to replace this hard coded value with the DNS name of the load balancer.

      So that's what we're going to do.

      We're going to update the launch template with some final configuration so that it can adjust this configuration replacing the IP address with the DNS name of the load balancer.

      So go back to the EC2 console, click on launch templates, select the WordPress launch template and click on actions modify template create new version.

      Under the template version description we're going to use app only, users EFS file system defined in /a4l/wordpress/efs/fsid and then ALB home added to the WP database.

      So we're going to make some on the fly adjustments to the WordPress database when every instance is provisioned to make sure that the load balancer DNS name is set to be the home URL for WordPress.

      So again scroll all the way down to the bottom because we're using an older template as the foundation for this one.

      All of the values will be pre-populated.

      Expand advanced details and scroll all the way down to user data and then just expand this text entry to make it slightly easier to interact with.

      As with the previous step position your cursor at the end of this top line and press enter twice.

      We need to add the first two lines of script which will bring in the application load balancer DNS name into an environment variable using systems manager parameter store.

      So now this instance when it's provisioning has the DNS name of the load balancer.

      Now next move all the way down to the bottom of this user data.

      So the last step that we want a machine to do when it's provisioning is to perform this update of the database.

      So there's a fairly large block of text which you need to copy from this stages text instructions.

      It's stage five and you need to paste this into the bottom of this file.

      So right at the bottom after these last two fine statements paste in this block.

      So this should start with the cat command on the top line of what you've just pasted in and then all the way down at the bottom.

      It should end with forward slash home forward slash EC2 hyphen user forward slash update underscore WP underscore IP dot SH.

      Essentially what this does is to bring in the WordPress configuration file to get the current authentication details for the database.

      So all these lines at the top are just designed to get the authentication information.

      So the DB name, the DB user and the DB password.

      This line runs a database script to get the old value for the IP address of the original IP address of the EC2 instance.

      So this is pulling in the original hard coded IP address.

      Then we're going to take the load balancer DNS name and we're going to run a series of SQL commands to update the database moving from that hard coded IP.

      To using the ALB DNS name.

      Now what this is actually doing is this line here is creating a script file and it's going to put into this script file everything until this EOF directive.

      So scrolling down this means that everything between these two lines is going to be stored in this script.

      Then we're going to make the script executable using CHmod 755.

      We're going to echo the path to this script into ETC RC.local which is run every time the instance is started up.

      And then finally we're going to run this script the once to update this information right here and now.

      So this new version of the launch template essentially changes what this hard coded IP address is every time to be the DNS name of the load balancer.

      It means if we ever change the DNS name of this load balancer this script will automatically correct this hard coded value.

      Now this is a thing specific to WordPress and there are many situations where you'll have applications which have certain nuances that you need to be aware of when creating elastic architectures.

      This is the one for WordPress.

      So now that we've made these changes go ahead and click on create template version to create that new version of this launch template.

      Click on launch template some for the final time we need to update the default version.

      So make sure this launch template is selected.

      Click on actions scroll down select set default version click in the drop down the current default version is version three we want to select version four so select that and then click set as default version.

      Now that means the launch template is updated and we can now provision instances in a fully elastic way.

      Okay so this is the end of part one of this lesson.

      It was getting a little bit on the long side and I wanted to give you the opportunity to take a small break maybe stretch your legs or make a coffee.

      Now part two will continue immediately from this point so go ahead complete this video and when you're ready I look forward to you joining me in part two.

    1. Welcome back to stage 4 of this advanced demo series.

      Now in stage 4, we're going to perform the last step before we can make this a truly elastic and scalable design.

      And we're going to migrate the wp-content folder which stores these priceless animal images from the EC2 instance onto EFS which is the elastic file system.

      This is a shared network file system that we can use to store images or other content in a resilient way outside of the life cycle of these individual EC2 instances.

      So to do that, we need to move back to the AWS console, click on the services drop down and type EFS.

      Right click and open the EFS console in a new tab.

      Once that's opened, click on create file system.

      Now we're going to step through the full configuration options so rather than using this simplified user interface, go ahead and click on customize.

      So the first step is to create the file system itself.

      So for name, go ahead and call this a4l-wordpress-content.

      Leave the storage class as standard.

      These cat images are critical data and so we are going to leave automatic backups enabled.

      And we're also going to leave life cycle management set to be the default so 30 days since the last access for throughput mode pick bursting which links the throughput to the size of the storage.

      Then expand additional settings.

      You've got two performance modes, general purpose and max IO.

      For this demonstration, go ahead and select general purpose.

      Max IO is for very specific high performance scenarios for 99% of use cases.

      You should select general purpose.

      Now also go ahead and untick enable encryption of data at rest.

      If this were a production scenario, you would leave this on.

      But for this demo, which is focusing on architecture evolution, it simplifies the implementation if we disable it.

      So go ahead and make sure that encryption is disabled.

      Once you've done that, that's all of the file system specific options that we need to configure.

      So go ahead and click on next.

      In this part, you're configuring the EFS mount targets, which are the network interfaces in the VPC, which your instances will connect with.

      So in the virtual private cloud drop down, select it and then pick a for L VPC.

      So this is the VPC that these mount targets are going to go into.

      Now, each of the mount targets is secured by a security group.

      The first thing we need to do is to strip off the default security group for the VPC.

      So click in the crosses next to each of these security groups.

      Now, you should have three rows, one for each availability zone.

      So in my case, you are seized one A, one B and one C and make sure that you've got the same selected.

      So one row for each availability zone, A, B and C.

      Now in the subnet drop down for availability zone one A, I want you to go ahead and pick SN-AP-A.

      So this should be 10.16.32.0/20.

      For the US East one B row, I want you to go ahead and pick SN-AP-B.

      This should be 10.16.96.0/20.

      And then finally for US East one C, I want you to go ahead and pick SN-AP-C, which should be 10.16.160.0/20.

      Now for all three rows within the security groups drop down, I want you to go ahead and select A4LVPC-SGEFS.

      Again, for each of these, it will have some randomness after it, but just make sure you pick the right one.

      A4LVPC-SGEFS.

      And you need to pick that for each of the three rows.

      Make sure you pick the right one because if you don't, it will impact your ability to connect.

      So there the mount targets configured and they'll be allocated with an IP address in each of these subnets automatically, which will allow you to connect to them.

      At this point, go ahead and click on Next.

      You can configure some additional file system policies.

      This is entirely optional.

      We won't be using that.

      So just go ahead and click on Next.

      And then on the review screen, scroll all the way down to the bottom and just click on Create.

      Now the file system itself will initially show as being in the creating state and it will then change to available.

      Go ahead and click on the file system itself.

      Click on the Network tab and then just scroll down and these are the mount targets which are being created.

      Now in order to configure our EC2 instance, we will need all of these mount targets to be in the available state.

      But what we can do to save some time is we can note down the file system ID of this EFS file system.

      So this is this value.

      You can see it at the top header here or you can see it in this row at the top.

      Just note that down and copy that into your clipboard because we need to configure another parameter to point at this file system ID.

      Because remember when we're scaling things automatically, it's always best practice to use the parameter store to store configuration information.

      So click on Services, type Sys which are the first few letters of Systems Manager and open that in a new tab.

      Once you're at the Systems Manager console, go ahead and click on Parameter Store and then you need to click Create Parameter to create a new parameter.

      We're going to call this parameter forward slash A4L forward slash WordPress forward slash and then EFS for Elastic File System, FS for File System and then ID.

      So EFS File System ID.

      For description, put File System ID for WordPress content and then in brackets WP-Content and that will help us know exactly what this parameter is for.

      As before, we'll be picking the standard tier, the type will be string, the data type will be text and then into the value, just go ahead and paste that file system ID.

      And once you've done all that, you can go ahead and click on Create Parameter.

      Once that's done, go back to the EFS console and if required, just hit refresh and make sure that all of these mount targets are in the available state.

      This is what it should look like with all three showing a green tick and available.

      Once that's the case, go to the EC2 console because now we're going to configure our EC2 instance to connect to this file system.

      So go to Running Instances, locate the WordPress -LT instance, right click, select Connect, choose Session Manager and then click on Connect.

      And this will open Session Manager console to the EC2 instance.

      As always, type shudubash, press Enter, cd and press Enter and then type clear and press Enter again, just to clear the screen making it easier to see.

      Now, even though EFS is based on NFS, which is a standard, in order to get EC2 instances to connect to EFS, we need to install an additional tools package.

      And to do that, we use this command.

      So type or paste that in and press Enter to install the EFS support package.

      Once that's installed again, I'm going to clear the screen to make it easier to see.

      Then I'm going to move to the Web Root folder by typing cd /vr/www/html.

      And what I'm going to do is to move the entire wp-content folder somewhere else.

      So if I just go inside this folder to illustrate exactly what it looks like and then do a list, you'll see that inside there are plugins, themes and uploads.

      And inside those folders are any media assets used by WordPress.

      So I'm just going to type cd /dot/ to move back up a level out of this folder.

      And then I'm going to move this entire folder to the /tmp folder, which is a temporary folder.

      So mv/wp-content///tmp and that moves that entire folder to the temporary folder.

      Then we're going to create a new folder.

      So shudu space mkdir space wp-content.

      This will be the mount point for the EFS file system.

      So I'm making an empty directory.

      Then I'm going to clear the screen and then paste in the next two commands from the lesson instructions.

      And this populates an environment variable called EFS/FSID with the value from the parameter you just created in the parameter store.

      So this is now the file system ID of the EFS file system.

      Now there's a file called fstab which exists in the /etc folder.

      And inside there it's called fstab and this contains a list of file systems which are mounted on this EC2 instance.

      Initially this only has the single line for the boot volume.

      What we're going to do is add an additional line to this fstab file.

      And this line is going to configure the EC2 instance so that it mounts our EFS file system on boot every single time.

      And this is this command.

      So it echoes this line.

      So the file system ID from the environment variable.

      We're going to mount it to the folder that we just created.

      So the wp-content folder and these are all of the file system options.

      So we're going to put that into the fstab file.

      So if we now cap this file it's got this extra line.

      And this means this file system will be mounted whenever the operating system starts.

      And we can force this just for now by running mount space-a space-t space-efs space-defaults.

      And this will mount the EFS file system onto this EC2 instance.

      We can verify that by doing a df space-k.

      And the bottom line should show us that we've now got this EFS file system mounted as the wp-content folder.

      So this is the folder that WordPress expects its media to be inside.

      Now all that remains is for us to migrate the existing data that we moved to the temporary folder back in to wp-content.

      And to do that we use this command.

      So we're using the mv command to move forward slash tmp forward slash wp-content forward slash star.

      So any files and folders and then we're moving it back into var www.html wp-content.

      So this is the EFS file system.

      So run that and that will copy the data back to EFS, which remember is now mounted where WordPress expects it to be.

      Now that might take a few moments to complete.

      Once it's done, we just need to fix up the permissions.

      So run this command chown space-bigr space-ec2-user colon apache space and then slash var slash www.

      So this just reestablishes permissions and ownership of everything in this particular part of the file system.

      Just make sure we won't have any problems going forward.

      Now at this point we're going to use the reboot command to restart this instance.

      And if everything goes well, the instance should start, the EFS file system should be loaded and WordPress should have access to all of this wp-content, which is now running from a network file system.

      So go ahead type reboot and press enter.

      If you press enter just to make sure that you are disconnected and I am.

      So that's good.

      So now I need to wait a few minutes for this EC2 instance or at least its operating system to restart.

      So I'll go ahead and close down this session manager tab.

      Go back to the EC2 console.

      After waiting a few minutes, I'll right click select connect check session manager click on connect.

      Assuming the instance has restarted, I'll be back at the prompt.

      And if I do a DF space-k if everything's working as expected, the EFS file system will still be mounted into the directory that we configured.

      If I go back to the EC2 console and just copy down the instances public IP version for address, either refresh the tab if you still got it open or paste in the IP address and reload that page.

      And if everything's working as expected, all of these high quality critical cat pitches should still load from the WordPress blog.

      So now at this point when we're interacting with the application, both the database and the wp-content both exist away from the EC2 instance.

      And this means we're now in a position where we can scale the EC2 instance without worrying about the data or the media for any of the posts.

      And this means we can now further evolve this architecture to be fully elastic.

      Now there is one more thing that we need to do before moving on to the next stage of the demo and implementing this final step towards a fully elastic architecture.

      And that's that we need to update the launch template to include this updated configuration so that it uses EFS.

      To do that, go back to the EC2 console, go to launch templates, select the launch template.

      So check the box, click on the actions drop down, select modify template, create new version.

      For template version description, use app only, uses EFS file system defined in and then the parameter store value that contains the file system ID.

      So this is just the description.

      Now again, because we're creating a new version, it will populate all of the configuration with the previous template version.

      But I'll need you to scroll all the way down to the bottom, expand advanced details and scroll all the way down.

      Again, we're going to make some edits to the user data.

      So expand this box a little bit to make it easier to read.

      What I'll need you to do is to put your cursor after the end of this top line and just press enter twice to make some space and then paste in this set of configuration.

      And again, this is stored within the instructions for this stage of the demo series that will just populate an environment variable with the file system ID that it will get from the parameter store.

      Scroll down and next you're looking for a software installation line.

      You're looking for this line, the line that performs the installation of the Maria DB server, the Apache web server and the W get utility.

      Position your cursor after the word stress and then press space.

      And then I'll want you to add this text followed by a space, which is Amazon hyphen EFS hyphen utils.

      Next, scroll down a little bit further and you're looking for the line that says system, CTL, start, HTTBD.

      Click on the end to position your cursor at the end of that line and then press enter twice to add some space and then paste in this next block also contained within this lessons instructions.

      What this does is to make a WP hyphen content folder before we install WordPress, configure the ownership of the entire folder tree and then add the line for EFS to the FSTAB file and then mount this EFS file system in to VARWWWW/HTML/WP hyphen content.

      And this means that when we're automatically provisioning this instance before we install WordPress, we're creating and mounting this EFS file system.

      And then we go on to installing WordPress, configuring the database and performing the final fix of all of the permissions at that folder structure.

      Next, scroll down.

      We're done with all of the launch template user data configuration.

      Just go ahead and click on create template version.

      We need to make this new version the default.

      So click on launch templates, select the WordPress launch template, click on actions, scroll down, select set default version, click in the dropdown.

      Version two should currently be the default.

      Change that to version three and click set as default version.

      So at this point, you further evolved the architecture.

      Now we have both the database for WordPress stored in RDS and the WP hyphen content data stored within the Elastic file system.

      So we've solved many of the applications limitations.

      We can scale the database independently of the application.

      We've stored the media files separate from the instance.

      So now we can scale the instance freely out or in without risking the media or the database.

      We do still have two final limitations which will be fixing together in the next stage of this demo series.

      One is that customers still connect to the instance directly so we don't have any health checks.

      We don't have any auto healing capabilities and we're limited to how we can scale.

      And then finally, the IP address of the instance is still hard coded into the database.

      And so even if we did provision additional instances, WordPress would expect all of the data to be loaded from that one single original instance.

      And to allow us to scale, we have to resolve both of those problems.

      At this point though, you've done everything required in stage four.

      So go ahead, complete this video.

      And when you're ready, I look forward to you joining me in stage five of this advanced demo series.

    1. Welcome back and in stage three of this demo series, you're going to change the single server architecture that's on screen now and move towards something a little more scalable.

      You're going to migrate the database from the EC2 instance into a separate RDS instance and that means each of these can scale independently, so you can grow or shrink the database independently of the EC2 instance.

      It also means that the data in the database lives past the lifecycle of the EC2 instance and this is required for later stages in the demo where you want to scale in and out based on load.

      So let's go ahead and do that.

      So you'll need to be at the AWS console, click on services and in the find services drop down, type RDS and then open that in a new tab.

      Now we're going to create a subnet group first and a subnet group is what allows RDS to select from a range of subnets to put its databases inside.

      In this case, we'll be giving RDS a selection of three subnets, so SN-DB-A, B and C.

      So three availability zones which it can choose to deploy database instances into.

      So to do that, look on the left hand menu and just click on subnet groups.

      Click on create DB subnet group.

      For name, call it WordPress RDS subnet group.

      Under description, just type RDS subnet group for WordPress.

      In the VPC drop down, select the A4L VPC.

      Scroll down a little and then under availability zones, click in the drop down and check the box next to US East 1A, 1B and 1C because we have database subnets in each of those availability zones and these were created as part of the infrastructure cloud formation template that you applied at the start of this advanced demo.

      Once you've selected those availability zones, next we need to pick the subnets inside those availability zones that the databases will go into.

      So click in the subnets drop down.

      Now you could go to the VPC console and get the IP address ranges that correspond to the different database subnets but I'm going to save us some time.

      So in US East 1A, you need to pick 10.16.16.0/20.

      That's the database subnet in availability zone A.

      In availability zone B, you need to pick 10.16.80.0/20.

      That's the database subnet in AZB.

      And then in US East 1C, you need to pick 10.16.144.0 because that's the database subnet in availability zone C.

      So now you've selected the three availability zones, the three subnets in those availability zones so you can scroll down and click on create.

      So that creates the database subnet group that RDS uses in order to select which subnets database instances should go into.

      The next step is to actually create the RDS instance itself.

      And to start with, we're going to use a free tier eligible database.

      So go ahead and click on databases, click on create database, select standard create.

      RDS is capable of using lots of different database engines, but we're going to select MySQL.

      So select MySQL.

      Scroll down and under version, put the version number that's inside this lesson's description.

      AWS regularly make changes and instead of using the version you see on this video, pick the one that's inside this lesson's description.

      Scroll down.

      Under templates, click on free tier because this will make sure that we're only selecting options that are eligible under the free tier.

      And we want to keep the first part of this demo series completely within the AWS free tier.

      Now under DB instance identifier, we need to give this instance a name.

      So delete this placeholder and then just enter A4L wordpress.

      Now for master username and password, we need to enter the values from the parameter store that we entered previously.

      So click on services, start typing sys and then right click on systems manager and open in a new tab.

      Go to the parameter store, look for the DB user parameter and then copy what's in the value field and then go back to the RDS console and paste that in for master username.

      So that should be A4L wordpress user.

      Do the same for the master password.

      So for that, you need to go back to parameter store and this time you're looking for A4L wordpress DB password.

      So select that.

      Once you're here, click on show and then copy the value for this parameter.

      Once you've got that value, paste it into both the master and confirm password boxes.

      Scroll down further still and now you need to pick the database instance size.

      Now because we've selected free tier eligible, we can only select DB.t3.micro.

      Or in some cases, this may be slightly different, but it's only going to allow you to pick free tier eligible instance types.

      So we can leave that selected.

      It is the default because we picked free tier only.

      Now scroll down to connectivity.

      Under the virtual private cloud VPC, click in the drop down and select the A4L VPC.

      So this defines the VPC that this database is going into.

      Once you've selected that, make sure for subnet group, you've got WordPress RDS subnet group selected.

      Choose no for publicly accessible and then for existing VPC security groups, I want you to go ahead and click on the cross next to default and then click in the drop down and select A4L VPC - SG database.

      And again, this will have some randomness on the end, but that's perfectly okay.

      So select A4L VPC - SG database.

      Under availability zone preference, select US East 1A.

      This makes sure this database just to start off with is in the same availability zone as the EC2 instance.

      Scroll down further still, go past database authentication and then expand additional configuration.

      And this is important because we need to set an initial database name.

      So for the initial database name, we'll need to go back to the parameter store.

      This time we need the value for the A4L WordPress DB name parameter.

      So select that and then copy its value.

      So copy that into the clipboard, go back to the RDS console and paste that in for the initial database name.

      And that should be A4L WordPress DB.

      At this point, we can leave everything else as default.

      So scroll all the way down to the bottom and click on create database.

      Now this can take anywhere up to 30 minutes to create the database and it will need to be fully ready before you move on to the next step.

      So now's a great time to pause this video, go and grab a coffee and wait for this database to become available, at which point you can resume the video.

      Now that this database instance is available, the next thing to do is to migrate the actual WordPress data.

      And to do that, we need to move back to the EC2 console.

      So open the EC2 console, locate WordPress -LT and then select that instance, right click, select connect, choose session manager and then click on connect.

      We're going to perform the migration from this instance itself.

      To start with run shudu space bash and press enter, cd and press enter and then type clear and press enter.

      We're going to be running some commands which are in the text instructions for this stage of the demo series.

      The first set of commands will load data from the parameter store into environment variables within the operating system.

      So go ahead and copy all of the first block of commands and paste it in to this terminal.

      This will load the DB password, the DB root password, DB user, DB name and DB endpoint all into environment variables and make sure to press enter on the last line just to complete that command.

      Next we're going to export the data from the local MariaDB database instance and we'll do that using this command.

      So mysqldump -h space and then uses these environment variables.

      So the database endpoint which will be local host and then a space -u and then a space and then the database user which is also an environment variable and then a space -p and then DB password which is an environment variable and then a space and then DB name which is also an environment variable.

      And then we direct the output of this command into a file called a4lwordpress.sql which is a database export file.

      So the best way is to copy and paste this out of the lesson instructions and then press enter and then run an ls space -la and just make sure that you've got that a4lwordpress.sql file and this is an output of the current sqldatabase for WordPress.

      Now next we need to change the parameter in parameter store for DB endpoint so that it points at our new RDS instance.

      So go back to the RDS console, click on the a4lwordpress instance and then copy this endpoint name into your clipboard.

      So it should start with a4lwordpress and then some random and then the region and then RDS and then amazonaws.com.

      So copy all of that into your clipboard and then either open the systems manager console and go to the parameter store or if you still got it open in a previous tab then you can open that tab.

      So click on parameter store to list all the parameters.

      Now at this point we're going to delete one of these parameters and it needs to be a deletion because we're going to recreate it.

      Please make sure that you do delete it and recreate it rather than just editing the value for the existing parameter because that won't work.

      You'll need to select the checkbox next to a4lwordpress.db endpoint and then click on delete.

      And once you've done that click on delete parameters to confirm that deletion and we're going to create a new parameter with the same name.

      So click on create parameter for name put forward slash a4lwordpress/db endpoint which is the same name as before.

      For description put WordPress endpoint name.

      We're going to use the standard tier again.

      It's going to be a string type.

      The data type is going to be text and then in the value paste in the RDS endpoint that you just copied into your clipboard.

      And once you've done that scroll down and click on create parameter.

      Go back to the session manager tab that you've got open to the instance and we need to refresh the environment variable with the updated parameter store parameter.

      So to do that copy and paste this next block of commands and this updates the db endpoint with the new RDS DNS name.

      Once we've updated that then we can run the mysql command to load in the a4lwordpress.sql export into the RDS instance and that's using this command.

      So again mysql -h space and then the RDS endpoint name which is in that environment variable and then specifying the db user db password and db name and then directing the command to load in the contents of this file.

      So if we paste all that in and press enter that imports that database export into RDS.

      So now RDS has the same data as our local Maria db installation.

      Now to finalize the migration we need to update the wordpress configuration file.

      So instead of pointing at the local Maria db instance it points at RDS.

      And we can do that using sed and perform a replace of local host with the contents of the db endpoint environment variable which remember now contains the DNS name for the RDS instance.

      And the location of the file that will be performing this replace on is /var/www/html/wp-config.php which is the wordpress configuration file.

      So paste that in and press enter and that's reconfigured wordpress now so that it talks to the RDS instance for the database functionality.

      Lastly we can run these commands to both disable Maria db so it doesn't start every time the operating system boots and set it to stopped right now.

      So now Maria db is no longer running on this EC2 instance.

      So we can verify that the functionality of our application is still there by going back to the EC2 console.

      Selecting wordpress -lt just copy this public IP address into your clipboard.

      If you already have it open in an existing tab you can refresh.

      It should still load the blog and yet we've still got the same best animals blog post.

      But now wordpress is loading the data for this blog post from the RDS instance.

      Now to be really clear at this point wordpress when you create a blog post has two different sets of data.

      It has the data of the blog post so the text, the metadata, the author, the date and time, the permissions, the published status and many other things they're stored in the database.

      But any media, any content for this blog post is still stored locally in a directory called wp-content.

      That is still on the EC2 instance or that we've migrated in this stage of the demo is the database itself from Maria db through to RDS.

      Now before we finish with this stage of the demo series there's one final task and that's to update the launch template so we can launch additional EC2 instances.

      But using this new configuration so pointing at the RDS instance.

      So to do that go back to the EC2 console and click on launch templates.

      Click in the checkbox next to the wordpress launch template.

      Select the actions drop down and then locate and click modify template create new version.

      Now for the description we're going to put single server app only.

      So we're indicating with this version of the launch template we no longer have the database inside the instance itself.

      Now because we're creating this from a previous version all of the boxes will be pre-populated.

      What we need to do is to update the user data.

      So go all the way down to the bottom and expand advanced details scroll all the way down to the bottom of that and find the user data box.

      And I find it's easier if we just expand it to make it slightly easier to see.

      There are a number of things which we need to adjust in this user data.

      First just scroll down and you need to locate this block of commands.

      So system CTL enable and system CTL start.

      What we need to remove are the lines that refer to MariaDB.

      So the top one is system CTL enable MariaDB select that and delete and then locate system CTL start MariaDB select that and delete.

      So that prevents MariaDB starting on the EC2 instance.

      Now because we're using an RDS instance we also need to remove this line which attempts to set the root password of the MariaDB database instance.

      We don't need that anymore so delete that.

      Scroll all the way down to the bottom and look for this block.

      So it starts with echo create database DB name and it finishes with RM/TMP/DB.setup.

      This is the block that creates the database within MariaDB, creates the user and sets all of the permissions.

      But because we're using RDS now we don't need to do any of this so we're going to delete this block as well.

      Once you've done that you can go ahead and click on create template version and this will create a new version but this time designed to use RDS.

      Once you've done that go back to the launch template screen and click on the launch template.

      We need to change it so that the new version is the default version that's used whenever we launch instances from this template.

      So click on the launch template.

      Once that's loaded you'll see we're currently on version one.

      Change this to version two and you'll see the updated details and then click on the actions drop down.

      Select set default version.

      In the dialogue make sure that version two is shown under template version and then click on set as default version.

      And at this point version two or the one which uses RDS is now set as default and this means when we use this template to launch any instances this is the version that will be used by default.

      Now at this point that's everything that I wanted you to do in stage three of this demo series.

      So you've migrated the data for a working WordPress installation from a local MariaDB database instance through to RDS.

      And that's essential to be able to scale this application because now the data is outside of the lifecycle of the EC2 instance.

      So we know that for any scale in or out events it won't impact the relational or SQL based data.

      It also means that we can scale the database independently of the WordPress application instances.

      So that helps us reach the desired outcome of a fully elastic architecture.

      Now at this point we've actually fixed many of the limitations of this design.

      At this point the only things that we need to fix are the application media.

      So the WordPress content which still resides in a folder local to the EC2 instance.

      So we need to migrate this out so that we can scale the instances in and out without risking that data.

      The other things that are still limiting factors are that customers are still connecting directly to the instance.

      So we need to resolve that by using a load balancer and the IP address of the instance is still hard coded into the database.

      So if this EC2 instance fails for whatever reason and we provision a new one, it won't function because WordPress expects everything to be loaded from this IP address.

      So that's something we need to resolve.

      But at this point that's everything you need to do in stage three.

      In stage four you'll be migrating these images from the EC2 instance into an elastic file system.

      And that's one of the last stages that we need to do before we can make this a fully elastic design.

      So go ahead complete this video and when you're ready I'll look forward to you joining me in stage four of this advanced demo series.

    1. Welcome back to stage two of this advanced demo lesson and again have included full instructions attached to this lesson.

      And this stage of the demo will be another one where you're entering lots of commands because you're going to automate the build of the WordPress application instance.

      So again, I would recommend opening the instructions for this demo lesson and copy and pasting the commands rather than typing them out by hand.

      Now at this point in the advanced demo series, you're going to have a leftover instance that you used to manually install WordPress in the previous stage.

      It should be called WordPress - Manual.

      So I'm going to want you to go ahead and right click on that and select terminate instance and confirm that process to remove this instance from your AWS account.

      We're going to be setting up exactly the same single instance deployment of WordPress, so both the database and the application on the same instance.

      But instead of manually building this, we're going to be using a launch template.

      So from the EC2 console, just go ahead and click on launch templates under instances.

      The first step is to create a launch template for our WordPress application.

      So go ahead and click on create launch template.

      Now launch templates are actually a new version of launch configurations that were previously used with auto scaling groups.

      Launch templates allow you to either launch instances manually using the template or they can be part of auto scaling groups.

      But what a launch template allows you to do is to specify all of the configuration in advance to launch an instance and that template can be used to launch one or many instances.

      So we're going to create a launch template which will automate the installation of WordPress, MariaDB and perform all of the configuration.

      And a launch template can actually have many different versions, which is a feature we'll use throughout this demo series as we evolve the design.

      So the first step is to name this template and we're going to call it WordPress.

      Under template version description, go ahead and enter single server DB and app.

      And then check this box which says provide guidance to help me set up a template that I can use with EC2 auto scaling.

      We're not immediately going to set it up as part of an auto scaling group, but it will help us highlight any options which are required if we want to use it with an auto scaling group.

      Now launch templates can actually be created from scratch or they can be based on a previous template version.

      If we expand source template, you're able to specify a template which this template is based on.

      But in this case, we're creating one from scratch so we won't set any of those options.

      Now just scroll down.

      So the next thing we're going to define in this launch template is the AMI that we're going to use.

      So go ahead and click on Quickstart.

      And once this has changed, we're going to use the same AMI we've been using previously.

      So I want you to go ahead and click on Amazon Linux, specifically Amazon Linux 2023.

      It should be the SSD volume type.

      It should be listed as free tier eligible and just make sure that you've got 64 bit x86 selected.

      And then scroll down further still and in the instance type drop down, we're looking for the T series of instances.

      And then you need to select the one that's free tier eligible.

      In most cases, this will be T2.micro, but select whichever is free tier eligible.

      We want to keep this advanced demo as much as possible within the free tier.

      Scroll down again and for key pair, just make sure that it says don't include in the launch template.

      Move down further still to network settings.

      Then make sure select existing security group is selected.

      And then in the security groups drop down, click in that and make sure that you select the A4L.

      VPC - SG WordPress.

      So this is the security group which will automatically be associated with any instances launched using this launch template.

      So select A4L.

      VPC - SG WordPress and there will be some randomness after this.

      That's fine.

      Just make sure you select the SG WordPress group and then we can scroll down further still.

      Now we can leave storage volumes as default.

      We won't set any resource tags.

      We won't do any configuration of network interfaces, but I will want you to expand advanced details.

      There are a few things that we need to set within advanced details.

      The first is an IAM instance profile.

      So click in this drop down and then make sure that you pick A4L.

      VPC - SG WordPress instance profile.

      Again, there will be some randomness.

      That's fine.

      What this is doing is creating the configuration which will attach an instance role to this EC2 instance.

      And this instance role is going to provide all the permissions required to interact with the parameter store and the elastic file system and anything else that this instance requires.

      And this was pre-created on your behalf using the cloud formation template.

      Next, scroll down further still and look for credit specification.

      Remember, this is the same option that you set when launching an instance manually.

      Now, as before, it's always best to set this to unlimited.

      But if you are using a brand new AWS account, then it's possible that AWS won't allow you to use this option.

      So you should probably go ahead and pick standard.

      It won't make that much of a difference.

      I'm going to pick unlimited, but I do suggest if you are using a fairly new account, you go ahead and select standard.

      So that's the configuration for the instance, the base level configuration.

      What I want you to do now though is to scroll all the way down to the bottom and there's a user data box.

      This user data allows us to specify bootstrapping information to automatically configure our EC2 instances.

      So into this user data box, I want you to paste the entire code snippet within stage 2B of this stages instructions.

      And again, they're attached to this lesson.

      The top line should be hash bang forward slash bin forward slash bash and then a space hyphen XE.

      And then if you scroll all the way down to the bottom, the last line should be RM space forward slash TMP forward slash DB dot setup.

      And now we can see we've pasted this entire user data.

      Once you've done that, go ahead and click on create launch template.

      Now that user data that you just pasted in is essentially all of the commands that you ran in the previous stage of the demo.

      Only instead of pasting them one by one, you've defined them within the user data.

      So this simply automates the process end to end.

      So to test this, go ahead and click on launch templates towards the top of the screen.

      It should show that you have a single launch template.

      It's called WordPress.

      The default version is one and the latest version is one.

      And as we move throughout this demo series, the latest version and the default version will change.

      So just keep an eye on those as we go.

      For now, though, I want you to click in the checkbox next to this launch template, click on actions and then launch instance from template.

      So this is going to launch an EC2 instance using this launch template.

      We're asked to choose a launch template and a version and define the number of instances and we can leave all of these as the defaults.

      If we just scroll down, you'll see how it's pre-populating all of these values with the configuration from the launch template.

      And that's what we want.

      Under key pair name, just select to proceed without a key pair not recommended.

      And that's the default value.

      Scroll down further still.

      Even the networking configuration is partially pre-populated.

      The only thing we need to do is specify a subnet that this instance will be launched into.

      And when we configure auto scaling groups to use this launch template, the auto scaling group will configure the subnets on our behalf.

      Because we're launching an instance directly from the launch template, we have to specify this subnet.

      So click in the subnet dropdown and then look for SN-PUB-A.

      Because we're going to deploy this WordPress instance into the public subnet in Availability Zone A.

      So select that.

      Scroll down.

      Look for the resource tag section and click on add tag.

      We're going to add a tag to the instance launched by this template.

      So into key, just type name and then for value, use WordPress-LT.

      And this will just tell us that this is an instance launched using the launch template.

      Once you've entered those, just scroll all the way down to the bottom and click launch instance.

      And this will launch an EC2 instance using this template.

      And this will automate everything that we had to do in the previous stage manually.

      So this saves us significant time and it enables us to use automation in later stages of this demo series.

      So now go ahead and click on the instance ID in this success box and this will take you to the EC2 console.

      Just give this instance a couple of minutes to finish its build process.

      Even though we're automating the process, it does still take some time to perform the installation and the configuration of all of those different components.

      So go ahead and just copy the public IP version for address of this instance into your clipboard.

      And then after you've waited a few minutes, open that in a new tab.

      If you get an error or it opens with a blank page, then you just need to give it a few minutes longer.

      But when it's finished, it should show the same WordPress installation screen.

      Once it does load the installation screen, we're going to follow the same process.

      So site title is Categorum, username is Admin.

      Enter the same password and then enter the fake test at test.com email address.

      Then click on install WordPress.

      Then click on login.

      Enter admin again.

      Enter the password.

      Click on login.

      It looks as though our automated WordPress build has worked because the dashboard has loaded.

      Click on posts.

      Delete the default post.

      Click on add new.

      For the title, the best animals again, click on the plus, select gallery, click on upload.

      And again, pick a selection of animal pictures and click on open.

      Remember, this is a new EC2 instance.

      So the one we previously terminated will have also deleted the data on that previous instance.

      Once these images have uploaded, click on publish and then publish again to upload the images to the EC2 instance and store the data within the database.

      So remember two components, the data stored in the database and the images or media stored locally on the EC2 instance.

      Click on view post to make sure that this loads correctly.

      It does.

      So that means the automatic build has worked okay.

      Everything's functioning as we expect.

      This has been an automatic build of a functional WordPress application.

      Now, the only thing that's changed from the previous stage of this advanced demo series is we've automated the build of this instance.

      It still has much the same limitations as the previous stage.

      So while we can improve the build time and we can use launch templates to support further automation, the database and application are still on the same instance.

      So neither can scale without the other.

      The database of the application is still located on that instance, meaning scale in or out operations risk this data.

      The WordPress content store is also stored locally on the instance.

      So again, any scale in or out operations risk the media that's stored locally as well as the database.

      Customers still connect directly to the instance, which means we can't perform health checks or automatically heal any failed instances.

      For this, we need a load balancer which we'll be looking at in later stages of this demo series.

      And of course, the IP address of the instance is still hard coded into the database.

      So this is something else we need to resolve as we move through the demo series.

      With that being said, though, that is everything that you needed to do in stage two of this demo series.

      So in this stage, you've automated the build of the WordPress instance using a launch template.

      Now, in stage three, you're going to migrate the data from the local database on EC2 into RDS.

      And this will move the data out of the lifecycle of the EC2 instance.

      And this makes it easier to scale.

      So in stage three, you're going to perform that migration and then update the launch template to take account of that configuration change.

      So go ahead and complete this stage of the demo lesson.

      And when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      Now that we've created all of those, we need to go ahead and install WordPress on our EC2 instance.

      So move back to instances.

      By now, the instance should be in the running state.

      Right-click, select Connect, change it to Session Manager, and then go ahead and click on Connect.

      This will allow us to connect into the EC2 instance without worrying about direct network access or having an SSH key pair.

      Once you're connected, go ahead and type shudu bash and press Enter, then type cd and press Enter, and then type clear and press Enter.

      And that will just clear the screen to make everything easy to see.

      Now, at this point, there are a lot of commands that you'll need to type in to manually install WordPress.

      Now, you can copy and paste these out of the text instructions for this stage of the demo lesson.

      But while you're doing so, I want you to imagine that you'd have to type these in one by one, because I want you to get an appreciation for just how long this install would take if you were doing it entirely manually.

      So first, we need to set some environment variables on this instance with the parameters that we've just stored in Parameter Store.

      So go ahead and copy all of this set of commands out of this stage's instructions, and this will set environment variables on this instance with values from the Parameter Store.

      And again, imagine how long this would take if you had to type all of this manually.

      Once we've got those variables configured, next we need to just update the operating system on the instance, make sure it's running with all the patches, and just update the package repositories.

      And we can do that with this command.

      The next set of commands in this stage's instructions install prerequisites.

      So this is the MariaDB database server, the Apache web server, WGet, some libraries, and a stress test utility.

      So go ahead and paste in the next block of commands to install all of these packages.

      Now, again, this is something that we will automate later in this demo series, but I want you to have an appreciation for just how long this takes.

      I'll type clear again to clear the screen, and then the next set of commands will start up the web server and the database server and ensure that both of them start up automatically when the instance operating system is first started.

      So if we restart this instance, both of these services will start up automatically.

      Again, make sure you press enter on the last command to make sure that starts up successfully.

      So that's the Apache web server and MariaDB that are both started and set to automatically start on operating system boot.

      Again, I'll clear the screen, and the next command that you'll run sets the root password for the MariaDB database server.

      So this is my SQL admin, and you're setting the password for the root user, and we're using the environment variable that we created earlier with values taken from the parameter store.

      So that sets the root password for the local database instance.

      Next, we're going to download and install WordPress, and we do that with the next block of commands.

      So this first downloads the WordPress package.

      It moves into the web root directory.

      It expands that package and then clears up after itself.

      So now we have WordPress installed.

      Again, I'll clear the screen to make it easier to see.

      This next set of commands replaces some placeholders in the wp-config.php file, which is the configuration for WordPress, and it replaces the placeholders with values taken earlier from the parameter store.

      So this is how we're configuring WordPress to be able to connect to the local MariaDB database server.

      The next block of commands that we use will fix up the permissions of all of this directory structure, so we don't have any problems accessing these files or any other security issues.

      Again, make sure you press Enter on the last command, and then we're almost done.

      The last step is to actually create the WordPress database, create the WordPress database user, set the password, and then grant permission on that database to that user.

      So these are all steps that we need to do because we're using a self-managed MariaDB database instance.

      So paste in this next block of commands and press Enter.

      So this has created a db.setup file with a number of SQL commands, and then it's used the MySQL utility to run those commands, which have created the database, the database user, and set permissions, and then it's cleared up the temporary file after all of that's been done.

      And at this point, that's all of the configuration needed.

      We've installed WordPress, we've installed MariaDB, we've started them both up, we've corrected permissions, and adjusted the configuration files.

      Now you've had the ability to copy and paste these commands from the lesson instructions, but imagine if you had to type them in all one by one.

      It would take much longer, and he's also something that's prone to many errors.

      That's something important to keep in mind as we move through this advanced demo.

      So the next step is to move back to the EC2 console.

      Make sure you've got the WordPress-manual instance selected, and then copy down the IP version for public IP address into your clipboard, and make sure that you do copy the public IP address, and don't click on the open address link, because that uses HTTPS, which we're not using.

      So go ahead and open that in a new browser tab.

      Now this is going to take you to the setup screen for WordPress.

      We're going to perform a quick setup.

      So under site title, I want you to enter CategorM.

      Under username, I want you to enter Admin.

      We'll keep things simple.

      For password, enter the Animals for Life password that we've been using in previous steps.

      Under email, go ahead and enter a fake email address, and then click on Install WordPress.

      That'll perform the final installation steps, at which point you can click on login.

      You'll need to enter the Admin username together with the password that you've just chosen, and then click on login.

      So this is the WordPress dashboard, and this suggests that our WordPress application is working absolutely fine.

      So to test it, just go to posts.

      We're going to delete the default post of Hello World.

      Once done, go ahead and click on Add New.

      You can just close down this Welcome to Block Editor dialog.

      Under title, use the best animal and then S, because we might have more than one animal, and then just put an exclamation mark at the end for effect.

      Click on the plus underneath that title, select Gallery.

      Click on Upload, select some animal pictures to upload.

      If you don't have any, you can go to Google Images and download some cat or dog or gerbil or guinea pig pictures.

      Anything that you want, chickens, snakes, just select a couple of animal pictures to upload, and then click on Open.

      And then once they've uploaded, you can go ahead and click on Publish, and then Publish again, and this will publish this post.

      And what it's doing in order to publish it is it's uploading the images into a local image store that's called wp-content.

      And in addition to that, it's storing the metadata for this post into the local MariaDB database.

      So there are two different places that data is stored, the local content store, as well as the database.

      So keep that in mind as we move on throughout this lesson.

      At this point, click on View Post.

      Just verify the post loads, it does.

      So that means everything's working as expected.

      Now, the configuration that you've just implemented has a number of important limitations.

      The first is that the application and database have been built manually, which takes time and doesn't allow automation.

      It's been slow and annoying, and that's very much the intention.

      Additionally, the database and the application are on the same instance.

      Neither of them can scale without the other.

      The database of the application is stored on an EC2 instance, and that means that scaling in or out risks data in this database.

      The application media, so the content is stored, also local to the instance in a folder called wp-content, and this means again, any scaling events in or out risks this media.

      Additionally, customer connections are directly to an instance, which prevents us from doing any form of scaling, automatic healing, or any health checks.

      One final part about WordPress that isn't commonly known is the IP address of the instance is actually hard-coded into the database.

      Now, where this starts to exhibit problems is when running inside AWS because EC2 instances don't have static IP addresses.

      If we go back to the EC2 console, right-click on this instance, and then stop the instance.

      Remember, a stop and start of an instance will not force the change of the public IP address of the instance, so restarting it isn't enough.

      You need to stop and then start.

      Watch what happens when the instance fully moves into a stop state.

      First, it loses this public IP address and it moves into the stop state.

      If I right-click to then start, that will take a few moments, but what will happen is once it's fully started, it will have a different IP version for public address.

      So now if I copy that IP address into my clipboard, move back to the tab where the website was previously open, and then open this new IP address in a different browser tab and note how it doesn't load.

      Even though the IP address is correct, it's not loading our WordPress website.

      The reason for that is the application is hard-coded with the IP address that was used to install WordPress.

      And so what it's attempting to do now is reference the old IP address.

      It's trying to contact the previous EC2 instance.

      Now, this is crucial because it prevents us from scaling the application.

      If we create new EC2 instances, they'll all point back at this instance.

      Even if we fix the database and content issues, we need to resolve the ability of WordPress to scale.

      And don't worry, we'll look at that later in this demo series.

      For now, that's everything you needed to do in stage one of this advanced demo.

      You've manually created a WordPress application with the application and database running on the same instance.

      In stage two, you're going to automate this process.

      So go ahead, complete this part of the demo series, and when you're ready, I'll look forward to you joining me in stage two.

    1. Welcome back and in this advanced demo lesson you're going to get the chance to experience how to do a practical architecture evolution.

      Now one of the things that I find very common amongst my students is that they complete the certification and as soon as they get their first job interview many of them which have an architectural scenario component they struggle on how to get started, how to design an architecture for a given scenario.

      So in this advanced demo series you're going to step through and evolve an architecture yourself.

      So you'll start with a single EC2 instance running the WordPress blogging engine and this single instance will be running the application itself, the database and it will be storing the content for all of the blog posts.

      And for this example we're going to assume it's an animal pictures blog.

      Now crucially in this first stage you're going to build this server manually to experience all of the different components that need to operate to produce this web application.

      Once you've built the instance manually next you'll replicate the process but using a launch template to provide automatic provisioning of this WordPress application but crucially it will still be the one single WordPress instance.

      Next you'll perform a database migration moving the MySQL database off the EC2 instance and running it on a dedicated RDS instance.

      So now the database, the data of this application will exist outside the life cycle of the EC2 instance and this is the first step of moving towards a fully elastic scalable architecture.

      Next once you've migrated the database instead of storing the content locally on the EC2 instance you'll provision the Elastic File System or EFS which provides a network based resilient shared file system and you'll migrate all of the content for the WordPress application from the instance to this Elastic File System.

      Once done these are all the components required to move this architecture to be fully elastic and that is being able to scale out or in based on load on that system.

      So the next step will be to move away from your customers connecting directly to this single EC2 instance.

      Instead you'll provision an autoscaling group which will allow instances to scale out or in as required and you'll configure an Elastic Load Balancer to point at that autoscaling group so your customers will connect in via the application load balancer rather than connecting to the instances directly and this will abstract your customers away from the instances it will allow your system to be fully resilient self-healing and fully elastically scalable.

      So by completing this advanced demo lesson you'll learn how to get started with scenario based questions as part of job interviews.

      With that being said let's go ahead and get started and to do that we need to move to the AWS console.

      To get started you're going to need to be logged in to a full AWS account without any restrictions you should be logged in as an admin user.

      If you're watching this demo as part of any of my courses then you need to use the general AWS account so that's the management account of the AWS organization which we've set up in the course and as always please make sure that you've selected the northern Virginia region.

      Now attached to this lesson are two links one of them is a one-click provision for the base infrastructure of this advanced demo lesson and the other is a link to the GitHub repository which contains text-based instructions for every stage of this advanced demo.

      So to start with go ahead and click on the one-click provisioning link.

      This is going to take you to the quick create stack page and everything should be pre-populated.

      The stack name should say A4LVPC all you need to do is check this acknowledgement box and then go ahead and click on create stack.

      Now you'll need to wait for this stack to move from create in progress to create complete before you can continue with the demo so go ahead and pause the video and you can resume it once this stack is in a create complete state.

      So now this stack's moved into the create complete state the first stage of this advanced demo series is to manually create a single instance WordPress deployment.

      Now this CloudFormation template has created the architecture that you can see on screen now so the VPC together with the three tier architecture so database application and public split across three different availability zones.

      So what you're going to do in this first part of this demo series is to create this single EC2 instance and you're going to do it manually so that you can experience all of the associated limitations.

      So make sure that you do have the text-based instructions open and the link for those is attached to this lesson because it will make it easier because you can copy and paste any commands or any configuration items.

      The first thing to do though is to click on services and then type EC2 into the services drop down and click on EC2 to move to the EC2 console.

      We're going to be launching our WordPress instance so what I need you to do is to click on launch instance and then again on launch instance.

      Now you should be fairly familiar with creating an EC2 instance so we're going to go through this part relatively quickly.

      So first you need to name the EC2 instance so go ahead and enter WordPress - manual in the name box and then scroll down and select Amazon Linux specifically Amazon Linux 2023 and just make sure that it's shown as free tier eligible.

      Simply make sure that it says 64-bit x86.

      Once set scroll down again and go to the instance type box, click in the drop down and just make sure that you have a free tier eligible instance selected.

      For most people this should be T2.micro but just make sure that it's an equivalent sized instance which is under the free tier.

      Continue scrolling down and under the key pair box just click in the drop down and select proceed without a key pair because we won't be connecting to this instance using an SSH key we'll be using session manager.

      Once selected scroll down further still and click on edit next to network settings.

      In the VPC box make sure that A4LVPC is selected.

      This is the animals for life VPC created by the one click deployment.

      Then under subnet make sure that SN-PUB-A is selected.

      This is the public subnet in availability zone A.

      Below this make sure that for both auto assign public IP and auto assign IPv6 IP both of these need to be set to enable.

      Once done scroll down again and next to firewall security groups check the box to say to select an existing security group.

      And then in the drop down make sure that you pick A4LVPC-SG WordPress.

      Now this will be followed by some randomness and that's okay just make sure that it's the SG-WordPress security group.

      This will allow us to connect into this instance using TCP port 80 which is HTTP.

      Once selected scroll down and we won't be making any changes to the storage we'll be using the default of 8GIB of GP3 storage.

      Below this expand advanced details and there are a couple of things that we need to change.

      First click on the drop down under IAM instance profile and just make sure that you select the A4LVPC-WordPress instance profile and again this will have some randomness after it and that's okay.

      Scroll down and next you're looking for a box which says credit specification.

      Now for this my preference is that you select unlimited because this will make the performance of the EC2 instance potentially better than not selecting anything at all or selecting standard.

      Now on brand new AWS accounts it's relatively common that you can't select unlimited.

      AWS don't allow you generally to select unlimited until the account has a billing history.

      So you might want to select standard here to avoid any problems.

      I'm going to select unlimited because my account allows it but if you've got a new AWS account then go ahead and select standard.

      If you do choose to select unlimited and you do receive an error then you can go ahead and repeat this process but select standard.

      So go ahead and select standard in your case and then scroll down and that's everything that we need to set at this point.

      Everything else looks good so go ahead and click on launch instance.

      So now that our instance is provisioning just go ahead and click on instances at the top and that will allow us to monitor the progress.

      Now we'll need this to be in a running state before we perform the WordPress installation but there's one more set of steps that I want to do first.

      Now throughout this advanced demo lesson we're going to be taking this single instance WordPress application and moving it towards a fully scalable or an elastically scalable design.

      Now to do that we need to move away from statically setting any configuration options so we're going to make use of the parameter store which is part of systems manager and we're going to create some parameters that our automatic build processes later in this demo will utilize.

      For now we're going to be performing everything manually but we'll still be using these variables because it will simplify what we have to type in the EC2 instance.

      So go ahead and click on services.

      Start typing systems manager and then once you see it populated in the list you can right click and open that in a new tab.

      Once you're at the systems manager console on the left under application management just locate parameter store and click it to move to the parameter store console and we're going to create a number of parameters.

      Now if you're watching this demo as part of my courses you may already have some parameters listed on this screen.

      If you have any existing ones which begin with forward slash A4L then go ahead and delete them before continuing.

      So go ahead and click on create parameter and the exact naming for each of these is in the full instructions contained on the github repository which is attached to this lesson so make sure you've got that open it'll make it significantly easier and less prone to errors.

      We're going to create a number of parameters for WordPress and the first is the database username so the username that will have permissions on the WordPress database.

      So I want you to set the name to forward slash A4L forward slash WordPress forward slash DB user.

      For description WordPress database user you can set the tier for the parameter to standard or advanced to keep things in the free tier we're going to use standard it's going to be a string parameter the data type is going to be text and the value needs to be our actual database username so for this demonstration we're going to use A4L WordPress user so enter that and click on create parameter.

      Now we're going to be moving quicker now now that you've seen the process our next parameter is going to be the database name so enter this in the name field for description WordPress database name again standard string data type of text and the value is going to be the WordPress database name so A4L WordPress DB scroll down and click on create parameter.

      Next is going to be the database endpoint so the host name that WordPress will connect to so for name enter this A4L WordPress DB endpoint for the description.

      WordPress endpoint name again standard string text for data type and then to start with because the database is on the same instance as the application the value will be local host so enter that and go ahead and click on create parameter.

      Next we'll be creating a parameter to store the password of the WordPress user so click on create parameter this time it's A4L WordPress DB password for description WordPress DB password again standard tier but this time it's going to be a secure string for KMS key source use current account and then for KMS key ID it will be alias AWS SSM which is the default KMS key for this service for value go ahead and enter a strong password again this is for the WordPress user that has permissions to access the database so if this were production it would need to be a strong password now I recommend that you use the same password as I'm using in this demo it uses number letter substitution and I know that it works with all of the different system components now I've included this password in the text based instructions and I do recommend that you use it in your demo as well go ahead and enter something in this value and then scroll down and click on create parameter and then last time click on create parameter again for name this time A4L WordPress DB root password and this is the root password for the local database server that's running on the EC2 instance so for description WordPress DB root password standard again and then again secure string because we're storing a password KMS key sources my current account leave everything else as default and then enter another strong password if this were production generally this would be different from the previous password but as this is a demo you should use the same strong password as you used previously whichever you choose go ahead and enter that into the value box and then click on create parameter okay so this is the end of part one of this lesson it was getting a little bit on the long side and I wanted to give you the opportunity to take a small break maybe stretch your legs or make a coffee now part two will continue immediately from this point so go ahead complete this video and when you're ready I look forward to you joining me in part two.

    1. Welcome back and in this demo lesson you're going to get the chance to quickly experience how session stickiness works with load balances.

      Now it's going to be a pretty brief demo lesson because I've tried to automate much of the infrastructure configuration that you've already done by this point in the course.

      I want to focus on this demo lesson purely on the session stickiness configuration so let's jump in and get started and we're going to start by applying a CloudFormation template which will create the basic infrastructure that we need.

      So I'm going to move across to the AWS console.

      Now to start with make sure you're logged into an AWS account and the user that you're using has admin privileges on that account and you've got the Northern Virginia region selected.

      Now attached to this demo lesson and in the demo instructions is a one click link that you can use to deploy the infrastructure so go ahead and click on that link.

      It'll take you to a quick create stack screen and all you'll need to do is to scroll all the way down to the bottom, check this capabilities box and then click on create stack.

      Now that can take anywhere from five to ten minutes to create so while that's creating let's talk through the architecture that you'll be using for this demo.

      The template which you're currently applying will create this architecture so it creates a VPC and then inside of that three public subnets one in each AZ then it creates an auto scaling group and linked to this is a launch template providing instance build directives.

      The auto scaling group is set to create six EC2 instances, two in each AZ and then it creates a load balancer configured to run from each public subnet.

      So this is the architecture that's going to exist in the AWS account once the cloud formation stack has finished creating.

      Now in this demo you're first going to connect to the load balancer with session stickiness disabled.

      This means that each time you connect to the load balancer the connection can be sent to any of the six instances meaning that each of them has around a 16.66 recurring chance to get a connection.

      So you'll first connect to the load balancer in this configuration.

      Once you've seen how that looks you're going to enable session stickiness and see how that affects the architecture.

      What will happen is the first time you connect to the load balancer with session stickiness enabled a cookie called AWS ALB will be generated and returned to your browser.

      Unfortunately for this guy it's not that type of cookie.

      What happens next is that any connections made while the cookie is valid are locked to one specific EC2 instance and they'll be locked to that instance until the cookie expires or that instance fails its health check at which point any connections will move to a different EC2 instance.

      Now at this point let's move back to the console and just check how the cloud formation creation process is going.

      At this point mine is still in a create in progress and you'll need this stack to be in a create complete state before moving on.

      So go ahead and pause the video and resume it once this changes to create complete.

      Okay so now the stack is in a create complete status we're good to move on and the first thing we'll need to do is verify that all of the six EC2 instances are functioning as they should be.

      So to do that go ahead and click on services and then type EC2 in the find services box and open that in a new tab then move to that tab and click on instances running.

      Now again this might look a little bit different in your account that's okay what we need to do is select each of these instances in turn and we're looking for the instance public IP version 4 DNS name.

      So go ahead and locate the public IP version 4 DNS field and just copy that into your clipboard and then open that in a new tab.

      The instance should load and it should show an instance ID, a random color background and an animated catgif.

      Now I want you to go ahead and open each of the remaining five instances each in its own tab so let's do that next.

      So select the second instance, scroll down, locate the public DNS address and then open that in a new tab.

      You'll see this has a different color background and a different animated catgif.

      We'll do the third instance, again different color background, different catgif.

      We'll do the fourth, once again different background, different gif.

      Do the fifth, different background, different gif and then finally the sixth instance.

      So we have each of the six EC2 instances all with a different background and a different catgif.

      Next scroll down on the menu on the left and click on load balances.

      You should see a load balancer which starts with ALB - ALB and then some random that's fine.

      Select that and copy the load balance the DNS name into your clipboard and then open that in a new tab.

      So this opens the load balancer and if you refresh that a few times you'll see that it moves between all of the EC2 instances.

      Now it could load the same instance twice or it might cycle through the same EC2 instances but you should see as you refresh it's cycling between all of the available instances and that's because we don't have session stickiness enabled.

      It's just doing a round robin approach to select different back-end instances within the target group.

      So each time we refresh there's a chance that it will move to a different back-end instance.

      Now let's assume at this point that we have an application which doesn't handle state in an external way so it stores the state on the EC2 instance itself.

      Well to enable session stickiness with application load balances we do it on a target group basis.

      So click on target groups and then click on the target group to go into its configuration.

      Locate and click on the attributes tab.

      Click on edit next to attributes.

      To enable session stickiness all we have to do is check this box select load balancer generated cookie and then pick validity period for the cookie that's generated by the application load balancer.

      So go ahead and leave this value as 1 but then click on the drop-down and change this from days to minutes.

      Once you've done that click on save changes and now session stickiness is enabled on this load balancer.

      If we go back to the tab that we have open to the load balancer and just keep hitting refresh you might notice initially that it changes to a new instance but at a certain point if you keep clicking it will lock to a specific EC2 instance and won't change.

      So now we're on this particular EC2 instance it's got this instance ID and even though we keep hitting refresh the background and the catgif remains the same.

      Now the way that this works and I can demonstrate this using Firefox if I go to the menu bar click on tools then browser tools then web developer tools and then click on the storage tab you'll be able to see that as part of accessing this load balancer I've got two cookies and one of the cookies the one that we're interested in is AWS ALB.

      This is the cookie that controls the session stickiness so every time I access this load balancer from the first point when this cookie is generated it passes this cookie back to the load balancer and it knows which back-end EC2 instance I should be connected to and so I will stay connected to this back-end instance until the cookie expires or this instance fails its health check.

      So let's test that what I want you to do is to copy down the instance ID that you're connected to and it will be different for you and just pay attention to the last few digits of the instance ID.

      Now if I go back to the EC2 console go to dashboard and then instances running locate the instance you just noted down the ID for right click and then stop that instance and confirm.

      We'll give that instance a few moments to stop if we go back to the load balancer tab and just keep hitting refresh now now that the instance is in a stopped state the load balancer detects that it's no longer valid and so I immediately switch to a brand new EC2 instance the cookie generated by the load balancer is updated to lock me to this new EC2 instance and I wouldn't have any idea that this back-end instance has failed and no longer responds to requests other than the fact that I can see that I've changed instances because I've created the instances to highlight which instance ID is being used.

      Now if I go back to the EC2 console select this instance again right click and this time start the instance even though this instance is started up again I won't reconnect to that original back-end instance because now I'm locked to this instance and there's a chance that what might happen while you're doing this demo lesson is while that instance was in a stopped state because this has been configured to use elastic load balancer health checks it might have detected that this instance is in a failed state and so it's instructed the auto scaling group to terminate that instance and replace it with a new one so don't be surprised if when you try to start this instance up it's in a terminated state that's okay the system is working as intended.

      So back to the load balancer tab I'll just keep hitting refresh and what we'll see is after the cookie expires there's always a chance that we could be moved onto a new EC2 instance.

      To return the configuration back to how it was at the start of the demo we can go back to the EC2 console go down to target groups open this target group click on the attributes tab and then edit and then uncheck the stickiness box and save the changes and at this point the cookie that's generated will no longer lock our connections to one specific back-end instance and so over time if we keep refreshing this page we should be moved between different back-end EC2 instances because again now we no longer have session stickiness.

      Now that's all I really wanted to highlight in this demo lesson I just wanted to give you some practical exposure to how the session stickiness feature works of application load balancers so this is something that you need to understand for the exam essentially if your application doesn't handle state externally to individual EC2 instances then you need the load balancer to make sure that any connections from a given user always end up on the same EC2 instance and the way to do that is with application load balancer controlled session stickiness now remember this does come with some negatives it means that the load balancer is not able to as efficiently distribute load across each of the back-end instances so while session stickiness is enabled it means customers are locked to one particular EC2 instance and even if customers locked to one instance generate much more load than customers locked to other instances the load balancer doesn't have the same level of flexibility to distribute connections so where possible application should be designed so they handle sessions externally to the instances and then you should not have session stickiness enabled and this is the way to ensure well-performing elastic architectures now at this point that's everything that you need to do in this demo lesson all that remains is to tidy up the environment so go back to the cloud formation console we can just go ahead and click on stacks click in the box next to the ALB stack click on delete and then click delete stack which will delete the stack and all of the infrastructure that it created at the start of this demo lesson at this point congratulations you've successfully completed this demo lesson and implemented the architecture that's on screen now as well as experienced how an application load balancer handles session stickiness so I hope you enjoyed the demo go ahead complete this video and when you're ready I look forward to you joining me in the next lesson.

    1. Welcome back and in this demo lesson you're going to experience the difference that EFS can make to our WordPress application architecture.

      Now this demo lesson has three main components.

      First we're going to deploy some infrastructure automatically using the one-click deployments.

      Then I'm going to step through the CloudFormation template and explain exactly how this architecture is built.

      And then right at the end you're going to have the opportunity to see exactly what benefits EFS provides.

      So to get started make sure that you're currently logged in to the general AWS account, so the management account of the organization, and as always you need to have the Northern Virginia region selected.

      Now this lesson actually has two one-click deployments.

      The first deploys the base infrastructure and the second deploys a WordPress EC2 instance, which has been enhanced to utilize EFS.

      So you need to apply both of these templates in order and wait for the first one to finish before applying the second.

      So we're going to start with the base VPC RDS EFS template first.

      So this deploys the base VPC, the Elastic File System and an RDS instance.

      Now everything should be pre-populated.

      The stack should be called EFS demo -vpc -rds -efs.

      Just scroll all the way down to the bottom, check the capabilities box and click on create stack.

      While that's going let's switch over to the CloudFormation template and just step through exactly what it does.

      So this is the template that you're deploying using the one-click deployment.

      It's deploying the Base Animals for Life VPC, an EFS file system as well as mount targets and an Aurora database cluster.

      So if we just scroll down we can see all of the VPC and networking resources used by the Base Animals for Life VPC.

      Continue scrolling down we'll see the subnets that this VPC contains IP version 6 information.

      We'll see an RDS security group, a database subnet group.

      We've got the database instance.

      Then we've got an instance security group which controls access to all the resources in the VPC that we use that security group on.

      Then we have a rule which allows anything with that security group attached to it to communicate with anything else.

      We have a rule that the WordPress instance will use and note that this includes permissions on the Elastic File System.

      Then we have the instance profile that that instance uses.

      Then we have the CloudWatch agent configuration and this is all automated.

      And if we just continue scrolling down here we can see the Elastic File System.

      So we create an EFS file system and then we create a file system mount target in each application subnet.

      So we've got mount target zero which is in application subnet A which is in US East 1A.

      We've got mount target one which is an application subnet B which logically is in US East 1B.

      And then finally target two which is in subnet app C which is in availability zone 1C.

      So we create the VPC, the database and the Elastic File System in this first one click deployment.

      Now we need this to be in a create complete state before we continue with the demo lessons.

      So go ahead and pause the video, wait for this to move into a create complete status and then we can use the second one click deployment.

      Okay that stacks now finished creating which means we can move on to the second one click deployment.

      Now there are actually two WordPress one click deployments which are attached to this lesson.

      We're going to use them both but for now I want you to use the WordPress one one click deployment.

      So go ahead and click on that link this will create a stack called EFS demo hyphen WordPress one.

      Everything should be pre-populated just go ahead and click on create stack.

      Now this is going to use the infrastructure provided by that first one click deployment.

      So it's going to use EFS demo hyphen VPC hyphen RDS hyphen EFS and let's quickly step through exactly what this is doing while it's provisioning.

      So this is the cloud formation template that is being used and we can skip past most of this.

      What I want to focus on is the resource that's being created so that's WordPress EC2.

      So this is using cross stack references to import a lot of the resources created in that first cloud formation stack.

      So it's importing the instance profile to use it's importing the web a subnet so it knows where to place this instance.

      And it's importing the instance security group that's created in that previous cloud formation stack.

      Now in addition to this if we look through the user data for this WordPress instance one major difference is that it's mounting the EFS file system into this folder.

      So forward slash var forward slash w w w forward slash HTML forward slash WP hyphen content.

      Now if you remember from earlier demo lessons this is the folder which WordPress users to store its media.

      So now instead of this folder being on the local EC2 file system this is now the EFS file system.

      The EFS file system is mapped into this folder on this WordPress instance.

      Other than that everything else is the same WordPress is installed.

      It's configured to you as the RDS instance the cow say custom login banner is displayed.

      It automatically configures the cloud watch agent and then it signals cloud formation that it's finished provisioning this instance.

      Now what we'll end up with when this stack has finished creating is an EC2 instance which will use the services provided by this original stack.

      So let's just refresh this.

      It's still in progress so go ahead and pause the video and wait for this stack to move into a create complete state and then we good to continue.

      So this stacks now finished creating and if we move across to the EC2 console so click on services locate EC2 right click and open that in a new tab.

      Then click on instances running and you'll see that we have this A4L WordPress instance.

      Now if we select that copy the IP address into your clipboard and then open that in a new tab we need to perform the WordPress installation.

      So go ahead and enter the site title the best cats and add some exclamation points.

      For username we need to use admin then for the password go back to the cloud formation stack and click on parameters and we're going to use the DB password.

      So copy that into your clipboard then go back paste it into the password box and then put test at test.com for the email address and click install WordPress.

      Then as before we need to log in so click on login admin for username reenter that password and click on login.

      Then we need to go to posts we need to click on trash below hello world to delete that post then click on add new close down this dialogue.

      For title put the best cats ever and some exclamation points then click on the plus click gallery click upload.

      There's a link attached to this lesson with four cat images so go ahead and download that link and extract it locate those four images select them and click on open.

      And then once you've done that click on publish and publish again and then click on view post.

      Now what that's doing in the background is it's adding these images to the WP hyphen content folder which is on the EC two instance but now we have that folder mounted using EFS and so the images are being stored on the elastic file system rather than the local instance file system.

      The cat pictures are there but what we're going to do to validate this is to go back to instances right click on this a four L hyphen WordPress instance and click on connect and then connect to this instance using EC two instance connect.

      Now once we connected to the instance you CD space forward slash VAR forward slash WWW forward slash HTML and then do an LS space hyphen LA to do a full listing you'll see that we have this WP hyphen content folder.

      So type CD space WP hyphen content and press enter then we'll clear the screen and do an LS space hyphen LA and then inside this folder we have plugins themes and uploads go into the uploads folder do an LS space hyphen LA depending on when you do this demo lesson you should see a folder representing the year so move into that folder then a folder representing the month again this will vary depending on when you do the demo lesson.

      Move into that folder and then you should see all four of my cat images and if you do a DF space hyphen K you'll be able to see that this folder so forward slash VAR forward slash WWW forward slash HTML WP hyphen content this is actually mounted using EFS so this is an EFS file system.

      Now this means the local instance file system is no longer critical it no longer stores the actual media that we upload to these posts so what we can do is we can go back to cloud formation go to stacks select the EFS demo hyphen WordPress one stack and then click on delete and delete that stack so that's going to terminate the EC two instance that we've just used to upload that media.

      We need to wait for that stack to fully delete before continuing so go ahead and pause the video and wait for this stack to disappear so that stacks disappeared and now there's a second WordPress one click deployment link attached to this lesson remember there are two so now go ahead and click on the second one this one should create a stack called EFS demo hyphen WordPress two scroll to the bottom and click on create stack that's going to create a new stack and a new EC two instance.

      So while we're doing this just close down all of these additional tabs at the top of the screen close them all down apart from the cloud formation one.

      We're going to need to wait for this to finish provisioning and move into the create complete state so again pause the video wait for this to change into create complete and then we go to to continue.

      After a few minutes the WordPress two stack has moved into a create complete state click on services open the EC two console in a new tab click on instances running you'll see a new A4L hyphen WordPress instance this is a brand new instance which has been provisioned using the one click deployment link that you've just used so the WordPress two one click deployment link.

      If we select this copy the public IP address into your clipboard and open that in a new tab it again loads our WordPress blog if we open the blog post.

      Now we can see these images because they're being loaded from EFS from the file system that EFS provides so no longer are we limited to only operating from a single EC two instance for our WordPress application because now there's nothing which gets stored specifically on that EC two instance.

      Instead everything stored on EFS and accessible from any EC two instance that we decide to give permissions to know what we can do to demonstrate this if we go back to cloud formation.

      Now remember attached to this lesson are two WordPress one click deployments we initially applied number one then we deleted that and applied number two so now I want you to reapply number one.

      So again click on the WordPress one one click deployment this again will create a new stack this time called EFS demo hyphen WordPress one click on create stack you need to wait for this to move into a create complete state so pause the video and resume it once the stack changes to create complete after a few minutes this stack also moves into create complete.

      Let's click on resources we can see it's provisioned a single EC two instance so let's click on this to move directly to this new instance select it copy this instance is IP address into your clipboard and open that in a new tab and again we have our WordPress blog and if we click on the post it loads those images so now we have a number of EC two instances we have to EC two instances both with WordPress installed both using the same RDS data.

      And both using the shared file system provided by EFS and it means that if any posts are edited or any images uploaded on either of these two EC two instances then those updates will be reflected on all other EC two instances and this means that we've now implemented this architecture that's on screen now and this is what's going to support us when we evolve this architecture more and add scalability in an upcoming section of the core.

      For now though we've just been focused on the shared file system now all that remains at this point is for us to tidy up the infrastructure that we've used in this demo lesson so close down all of these tabs we need to be at the cloud formation console we need to start by deleting EFS demo WordPress one and WordPress two so pick either of those click delete and then delete stack then select the other delete and then delete stack.

      Now we need both of these to finish deleting and then we can delete this last stack so go ahead and pause the video wait for both of these to disappear and then we can resume both of those have deleted so now we can click the final stack EFS demo hyphen VPC hyphen RDS hyphen EFS so select that delete and then delete stack and that's everything that you need to do in this demo lesson and once that stacks finished deleting the account will be in the same state as it was at the start of this.

      Now I hope you've enjoyed this demo lesson and that it's been useful what you've implemented in this demo is one more supportive step towards us moving this architecture from being a monolith through to being fully elastic.

      Now the application is in this state where we have a single shared RDS database for all of our application instances and we're also using a shared file system provided by EFS and this means that we can have one single EC2 instance we could have two EC2 instances or even 200 all of them sharing the same database and the same shared file system provided by EFS.

      Now in an upcoming section of this course we're going to extend this further by creating a launch template which automatically builds EC2 instances as part of this application architecture.

      We're going to utilize auto scaling groups together with application load balancers to implement an architecture which is fully elastic and resilient and this has been one more supportive step towards that objective.

      At this point though that's everything that you needed to do in this demo lesson so go ahead complete this video and when you're ready I look forward to you joining me in the next.

    1. Welcome back, this is part two of this lesson.

      We're going to continue immediately from the end of part one, so let's get started.

      Okay, so all three of these mount targets are now in an available state and that means we can connect into this EFS file system from any of the availability zones within the Animals for Life VPC.

      So what we need to do is test out this process and we're going to interact with this file system from our EC2 instances.

      So move back to the tab where we have the EC2 console open.

      And at this point I want you to either, and this depends on your browser, I'll either want you to right click and duplicate this tab to open another identical copy.

      If you can't do this in your browser then just open a new tab and copy and paste this URL into that tab.

      You'll end up with two separate tabs open to the same EC2 screen.

      So on the first tab we're going to connect to A4L-EFS instance A.

      So right click and then select connect.

      We're going to use instance connect.

      So make sure the username is EC2-user and then click on connect.

      Now right now this instance is not connected to this EFS file system and we can verify that by running a DF space-k and press enter.

      You'll see that nowhere here is listed this EFS file system.

      These are all volumes directly attached to the EC2 instance and of course the boot volume is provided by EBS.

      Now within Linux all devices or all file systems are mounted into a folder.

      So the first thing that we need to do to interact with EFS is to create a folder for the EFS file system to be mounted into.

      And we can do that using this command so shudu space mkdir space-p space/efs/wp-content.

      Now the hyphen p option just means that everything in this path will be created if it's not already.

      So this will create forward/EFS if it doesn't already exist.

      So press enter to create that folder.

      So I'm going to clear the screen to keep this easy to see.

      And the next thing I need to do is to install a package of tools which allows this instance or specifically the operating system to interact with the EFS product.

      Now the command I'm going to use to install these tools is shudu to give us admin permissions and then DNF which is the package manager for this operating system.

      And then a space hyphen y to automatically acknowledge any prompts and then a space and then install because I want to install a package and then a space.

      And then the name of the tools that I want to install is amazon hyphen EFS hyphen utils.

      So this is a set of tools which allows this operating system to interact with EFS.

      So go ahead and press enter and that will install these tools and then we can configure the interaction between this operating system and EFS.

      Again I'm going to clear the screen to keep this easy to see and I want to mount this EFS file system in that folder that we've just created.

      But specifically I want it to mount every time the instance is restarted.

      So of course that means we need to add it to the FSTAB file.

      Now if you remember this file from elsewhere in the course it's contained within the forward/ETC folder.

      So we need to move into that folder cd///ETC and then the file is called FSTAB.

      So we need to run shudu to give us admin permissions and then nano which is a text editor and then the name of the file which is FSTAB.

      So press enter and the file will likely have only one or two lines which is the root and/or boot volume of this instance.

      So let's just move to the end because we're going to add a new line and this is contained within the lesson commands document but we're going to paste in this line.

      So this line tells us that we want to mount this file system ID so file system ID colon forward/.

      We want to mount that into this folder so forward/efs forward/wp-content.

      We tell it that the file system type is EFS.

      Remember EFS is actually based on NFS which is the network file system but this is one provided by AWS as a service and so we use a specific AWS file system which is EFS.

      And the support for this has been installed by that tools package which we just installed.

      Now the exact functionality of this is beyond the scope of this course but if you do want to research further then go ahead and investigate exactly what these options do.

      What we need to do though is to point it at our specific EFS file system.

      So this is this component of the line all the way from the start here to this forward/.

      So to get the file system ID we need to go back to the EFS console and we need to copy down this full file system ID and yours will be different so make sure you copy your own file system ID into the clipboard.

      Then go back here and select the colon and then delete all the way through to the start of this line.

      And once you've done that paste in your file system ID what it should look like is the file system ID then a colon and then a forward/.

      So at this point we need to save this file so control O to save and then enter and control X to exit.

      Again I'm going to clear the screen to make it easier to see.

      Then I'll run a DF space -K and this is what the file systems currently attached to this instance look like.

      Then we're going to mount the EFS file system into the folder that we've created and the way that we do this is with this command.

      So shudu mount and then we specify the name of the folder that we want to mount.

      Now the way that this works is that this uses what we've just defined in the FSTAB file.

      So we're going to mount into this folder whatever file system is defined in that file.

      So that's the EFS file system and if we press enter after a few moments it should return back to the prompt and that's mounted that file system.

      There we go we back at the prompt and if we do a DF space -K again we'll see that now we've got this extra line at the bottom.

      So this is the EFS file system mounted into this folder.

      Now to show you that this is in fact a network file system let's go ahead and move into that folder using this command.

      And now that we're in that folder we're going to create a file.

      So we're going to use shudu so that you have admin privileges and then we're going to use the command touch which if you remember from earlier in the course just creates an empty file.

      And we're going to call this file amazing test file dot txt.

      Go ahead and press enter and then do an LS space -LA and you'll see that we now have this file created within this folder.

      And while we're creating it on this EC2 instance it's actually put this file on a network file system.

      Now to verify that let's move back to the other tab that we have open to the EC2 console the one that's still on this running instances screen.

      And now let's go ahead and connect to instance B.

      So right click on instance B select connect again instance connect verify the username is as it should be and click on connect.

      So now we're on instance B.

      Let's do a DF space -K to verify that we don't currently have any EFS file system mounted.

      Next we need to install the EFS tools package so that we can mount this file system.

      So let's go ahead and install that package clear the screen to make it easier to see then we need to create the folder that we're going to be mounting this file system into.

      We'll use the same command as on instance A.

      Then we need to edit the FSTAB file to add this file system configuration.

      So we'll do that using this command so shudu space nano space forward slash ETC forward slash FSTAB press enter.

      Remember this is instance B so it won't have the line that we added on instance A.

      So we need to go down to the bottom paste in this placeholder and then we need to replace the file system ID at the start with the actual file system ID.

      So delete this leaving the colon and forward slash go back to the EFS console copy the file system ID into your clipboard.

      Move back to this instance paste that in everything looks good.

      Save that file with control O press enter exit with control X then we back at the prompt clear the screen.

      We'll use the shudu mount forward slash EFS forward slash WP hyphen content command again to mount the EFS file system onto this instance and again it's using the configuration that we've just defined in the FSTAB file press enter.

      After a few moments you'll be placed back at the prompt we can verify whether this is mounted with DF space hyphen K.

      It has mounted by the looks of things it's at the bottom.

      So now if we move into that folder so CD forward slash EFS forward slash WP hyphen content forward slash and press enter.

      We now in that folder and if we do a listing so LS space hyphen LA what we'll see is the amazing test file dot txt which was created on instance A.

      So this proves that this is a shared network file system where any files added on one instance are visible to all other instances.

      So EFS is a multi user network based file system that can be mounted on both EC2 Linux instances as well as on premises physical or virtual servers running Linux.

      Now this is a simple example of how to use EFS for now we've done everything that we need to do in this demo lesson so we just need to clean up all of the infrastructure that we've used to do that.

      Go back to the EFS console we're going to go ahead and delete this file system so we should already have it selected just select delete you'll need to confirm that process by pasting in the file system ID.

      So go ahead and put your file system ID and then select confirm.

      Now that can take some time to delete and you'll need to wait for this process to complete.

      Once it has completed we're going to go ahead and move across to the cloud formation console.

      You should still have this open in a tab if you don't just type cloud formation in the search box at the top and then move to the cloud formation console.

      You should still have the stack name of implementing EFS which is the stack you created at the start with the one click deployment.

      Go ahead and select this stack then click on delete and confirm that deletion and once that finishes deleting that's all of the infrastructure gone that we've created in this demo lesson.

      So I hope this has been a fun and enjoyable demo lesson where you've gained some practical experience of working with EFS at this point though that is everything that you need to do in this demo lesson.

      So go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome back and in this demo lesson I want to give you some abstract practical experience of using the Elastic File System or EFS.

      Now we're going to need some infrastructure.

      Before we apply that as always make sure that you're logged into the general AWS account, so the management account of the organization and you'll need the Northern Virginia region selected.

      Now attached to this lesson is a one-click deployment link so go ahead and click that.

      This is going to provision some infrastructure.

      It's going to take you to the quick create stack screen and everything should be pre-populated.

      You'll just need to scroll to the bottom, check the box beneath capabilities and then click on create stack.

      You're also going to be typing some commands within this demo lesson so also attached to this lesson is a lesson commands document.

      Go ahead and open that in a new tab.

      So this is just a list of the commands that we're going to be using during the demo lesson and there are some placeholders such as file system ID that you'll need to replace as we go but make sure you've got this open for reference.

      Now we're going to need this stack to be in a create complete state before we continue with the demo lesson so go ahead pause the video and resume it once your stack moves into a create complete state.

      Okay so the stacks now moved into a create complete status and what this has actually done is create the animals for life base VPC as well as a number of EC2 instances.

      So if we go to the EC2 console and click on instances running you'll note that we've created a for L - EFS instance A and a for L - EFS instance B and we're going to be creating an EFS file system and mount points and then mounting that on both of these instances and interacting with the data stored on that file system.

      We're going to get you the experience of working with a network shared file system so let's go ahead and do that.

      So to get started we need to move to the EFS console so in the search box at the top just type EFS and then open that in a brand new tab.

      We're going to leave this tab open to the instances part of the EC2 console because we're going to come back to this very shortly.

      So let's move across to the EFS console that we have open in a separate tab and the first step is to create a file system so a file system is the base entity of the elastic file system product and that's what we're going to create.

      Now you've got two options for setting up an EFS file system you can use this simple dialogue or you can click on customize to customize it further.

      So if we're using the simple dialogue we'd start by naming the file system so let's say we use A4L - EFS and then you'd need to pick a VPC for this file system to be provisioned into and of course we'd want to select the animals for life VPC.

      Now we want to customize this further we don't want to just accept these high-level defaults so we need to click on customize.

      This is going to move us to this user interface which has many more options so we've still got the A4L - EFS name for this file system.

      Now for the storage class we're going to pick standard which means the data is replicated across multiple availability zones.

      If you're doing this in a test or development environment or you're storing data which is not important then you can choose to use one zone which stores data redundantly but only within a single AZ.

      Now again in this demonstration we are going to be using multiple availability zones so make sure that you pick standard for storage class.

      You're able to configure automatic backups of this file system using AWS backup and if you're taking an appropriate certification course this is something which I'll be covering in much more detail.

      You can either enable this or disable it obviously for a production usage you'd want to enable it but for this demonstration we're going to disable it.

      Now EFS as I mentioned in the theory lesson comes with different classes of storage and you can configure lifecycle management to move files between those different storage classes so if you want to configure lifecycle management to move any files not accessed for 30 days you can move those into the infrequent access storage class and you can also transition out of infrequent access when anything is accessed so go ahead and select on first access for transition out of IA.

      So in many ways this is like S3 with the different classes of storage for different use cases.

      When you're creating a file system you're able to set different performance and throughput modes.

      For throughput mode you can choose between bursting and enhanced.

      If you pick enhanced you're able to select between elastic and provisioned.

      I've talked more about these in the theory lesson.

      We're going to pick bursting.

      Now for performance you can choose between general purpose and max I/O.

      General purpose is the default and rightfully so and you should use this for almost all situations.

      Only use max I/O if you want to scale to really high levels of aggregate throughput and input output operations per second so only select it if you absolutely know that you need this option.

      You've also got the ability to encrypt the data on the file system and if you do encrypt it it uses KMS and you need to pick a KMS key to use.

      Of course this means that in order to interact with objects on this file system permissions are needed both on the EFS service itself as well as the KMS key that's used for the encryption operation.

      Now this is something that you will absolutely need to use for production usage but for this demonstration we're going to switch it off.

      We won't be setting any tags for this file system so let's go ahead and click on next.

      You need to configure the network settings for this file system so specifically the mount targets that will be created to access this file system.

      Now best practice is that any availability zones within a VPC where you're consuming the services provided by EFS you should be creating a mount target so in our case that's US - East - 1A, 1B and 1C.

      So we're going to go through and configure this so first let's delete all of these default security group assignments.

      Every mount target that you create will have an associated security group so we'll be setting these specifically.

      For now though we need to choose the application subnet in each of these availability zones so in the top drop-down which is US - East - 1A I'm looking for app A so go ahead and do the same.

      In US - East - 1B I want to select the app B subnet and then in US - East - 1C logically I'll be selecting the app C subnet so that's app A, app B and app C.

      Now for security groups the CloudFormation 1 click deployment has provisioned this instance security group and by default this security group allows all connections from any entities which have this attached so this is a really easy way that we can allow our instances to connect to these mount targets so for each of these lines go ahead and select the instance security group you'll need to do that for each of the mount targets so we'll do the second one and then we'll do the third one and that's all of the network configuration options that we need to worry about so click on next it's here where you can define any policies on the file system so you can prevent root access by default you can enforce read only access by default you can prevent anonymous access or you can enforce encryption in transit for all clients connected to this EFS file system so any clients that connect to the mount targets to access the file system you can ensure that that uses encryption in transit and if you're using this in production you might want to select at least this last option to improve security for this demo lesson we're not going to use any of these policy options nor are we going to define a custom policy in the policy editor instead we'll just click on next at this point we just need to review everything's to our satisfaction everything looks good so we're going to scroll down to the bottom and just click on create now in order to continue with this demo lesson we're going to need both the file system and all of its mount targets so go into the file system click on network and you'll see three mount targets being created all three of these need to be ready before we can continue the demo lesson so this seems like a great time to end part one of this demo lesson go ahead and finish this video and then when all of these mount targets are ready to go you can start part two.

    1. Welcome back and in this demo lesson you're going to get some experience of migrating a snapshot which you've previously taken from Aurora in provisioned mode and migrate this into Aurora running in serverless mode.

      Now before we begin as always make sure that you're logged in to the general AWS account, so the management account of the organization and you'll need to have Northern Virginia selected.

      Now attached to this lesson is a one-click deployment link and I'll need you to go ahead and open that to start the process.

      Now once you've got this open we're going to need the Aurora snapshot name that you created in a previous demo.

      So click on the services drop down and locate RDS.

      It's probably going to be in the recently visited section if not you can search in the box at the top but go ahead and open that in a new tab.

      Go to that tab and once it's loaded click on snapshots and you should have these two snapshots in your account.

      The first snapshot is a4l wordpress -with -cat - post -mySQL57.

      The other snapshot the one that we're going to use is this one so a4l wordpress -aura -with -cat -post.

      Go ahead and select that entire snapshot name and copy that into your clipboard because we're restoring an Aurora provisioned snapshot into Aurora serverless.

      We're not performing a migration we're performing a restore and so we don't need the snapshot ARN we need the snapshot name.

      So go back to the cloud formation stack everything should be pre-populated but there's a box that you need to paste in the snapshot name that you'll be restoring so that's this one paste that in check the acknowledgement box at the bottom under capabilities and then create stack.

      Now that process can take up to 45 minutes to complete sometimes it can be a little bit quicker and while that's working we're going to follow the same process through manually but we're going to stop before provisioning the Aurora serverless cluster.

      So go back to the RDS tab make sure that you still have this snapshot selected then click on actions and then restore snapshot and I want to step through the options available when restoring an Aurora provisioned snapshot into Aurora serverless.

      So these are the options you'll have when you're restoring an Aurora provisioned snapshot you'll see a list of compatible engines so anything compatible with the snapshot that you're restoring in our case it's only my SQL compatibility then you'll have to select your capacity type now it defaults to provisioned but we want to restore to a serverless cluster so we'll select serverless.

      You need to select the version of Aurora serverless that you're restoring to and again it's only going to show you compatible versions in this case only 2.07.1 and that's why I was so precise with the version numbers when doing the demos earlier in this section.

      Now under database identifier it's here where we would need to provide a unique identifier within this region inside this account for what we're restoring so we might use a4l wordpress-serveless we then need to provide connectivity information so we'd click in the VPC drop-down and make sure we select the animals for life VPC we'd still need to provide a database subnet group to use now currently there isn't one that exists in the account because the cloud formation template is still provisioning but we'd need to choose a relevant subnet group in this box we'd also need to choose a VPC security group which controls access to this database cluster then we have additional configuration and this is a feature which I'm going to be talking about in a dedicated lesson if you're doing the developer or sysops associate courses and this is an API which can be provisioned to give access to the data within this Aurora serverless cluster and it can do so in a way which is very lightweight and this makes it ideal for use with things like serverless applications which prefer a connectionless architecture so this is something that you will use if you want to use for example Aurora serverless with a serverless application based on lambda now something unique to Aurora serverless is the concept of capacity units and I've talked about these in the theory lesson where I talk about Aurora serverless these are the units of database service which the Aurora serverless cluster can make use of and you're able to set a minimum capacity unit and a maximum capacity unit and this provides a range of resources that this cluster can move between based on load on the cluster so as I've talked about in the theory lesson it will automatically provision more capacity or less capacity between these two values based on load now you have additional options for scaling and one that I'll be demonstrating a little bit later on in this demo lesson is how you can actually pause the compute capacity after a consecutive number of minutes of inactivity and this as long as your application supports it can actually reduce the amount of cost that you have running a database platform down to almost zero so you won't have any compute capacity build when the Aurora serverless cluster isn't in use and again I'll be demonstrating that very shortly in this demo lesson you're able to set encryption options just like with other forms of RDS and then under additional configuration you can also configure backup options now these options are obviously based on restoring a snapshot and you have a similar yet more extensive set of options if you're creating an Aurora serverless cluster from scratch so if we select Amazon Aurora and then we go down and select the serverless capacity type then obviously we can select from different versions and we have a wider range of options that we can set so the cluster identifier the admin username and password we've still got the capacity settings we still need to define connectivity options we've got additional configuration options around creating a database controlling the parameter group customizing backup options encryption and enabling deletion protection so whether you're restoring a snapshot or creating an Aurora serverless cluster from scratch these options are similar but you have access to slightly more configuration if you're creating a brand new cluster because when you're restoring a snapshot many of these configuration items are taken from that snapshot at this point we're not going to actually create the cluster manual so I'm going to cancel out of that and I'm going to refresh and as you can see we already have our Aurora serverless DB cluster and it's in an available state so let's go back to our cloud formation stack and refresh it's still in a create in progress state for the stack itself and in order to continue with this demo lesson we're going to need this to be in a create complete state so go ahead pause the video wait for your stack to move into a create complete state and then we can continue so this stacks now moved into a create complete state and we're good to continue so the first thing that I want to draw your attention to if we move back to the RDS console and then let's just refresh you'll see that this cluster is currently using two Aurora capacity units let's go inside the cluster we'll be able to see that it's available it's currently using two capacity units but otherwise it looks very similar to a provisioned Aurora cluster now what we're going to do is click on services open the EC2 console in a new tab go to instances running you should see a single WordPress instance so select that copy the public IP version 4 address into your clipboard making sure not to use this open address and open that in a new tab you'll see that it loads up the WordPress application and it still has the post within it that you created in the previous demo lesson the best cats ever and if you open this post you'll see that it doesn't have any of the attached images because remember they're not stored in the database they're stored on the local instance file system and that's something that we're going to rectify in an upcoming section of the course either called advanced storage or network storage depending on what course you're currently taking but I just wanted to demonstrate that all we've done is restore an Aurora provision snapshot into an Aurora serverless cluster and it still operates in the same way as Aurora provisioned but this is where things change if we go back to the RDS console we know that this Aurora serverless cluster makes use of Aurora capacity units or ACUs and currently it's set to move between one and two Aurora capacity units and the reason it's currently set to two is because we've just used it we've just restored an existing snapshot into this cluster and that operation comes with a relatively high amount of overhead so it needs to go to the two capacity units maximum in order to give us the best performance now what we're going to see over the next few minutes if we just sit here and keep refreshing this screen what should happen is that because we're not using our application first we're going to see it drop down from two capacity units to one capacity unit and that will of course reduce the costs for running this Aurora serverless cluster after a certain amount of time it's going to go from one capacity unit to zero capacity units because it's going to pause the cluster because of no usage we've got this configured if I click on the configuration tab to pause the compute capacity after a number of consecutive minutes of inactivity and it's set to five minutes so after five minutes of no usage on this database it's actually going to pause the compute capacity and we won't be incurring any costs for the compute side of this Aurora serverless cluster so that's one of the real benefits of Aurora serverless versus all of the other types of RDS database engine so let's just go ahead and refresh this and see if it's already changed from two capacity units it's currently still on two so let's select logs and events and refresh we don't see any events currently so this means that we've had no scaling events on this database but if we click on monitoring you'll see how the CPU utilization has decreased from around 25% to just over 5% and the database connection count has reduced from the one when we just accessed the application back down to zero after a few refreshers we'll see that it either decreases from two capacity units down to one or it will go straight to zero if we reach this five minute timer before it performs that scaling event to reduce from two to one so in our case we've skipped the point of having one capacity unit we've reached that five minute threshold where it pauses the compute capacity and so it's gone straight down to zero so your experience might vary it might go from two down to one and then pause or it might go from two straight down to zero but in a case for me my database is currently running at zero capacity units because this time frame has been reached with no activity and the compute has been paused so this means I have no costs for the compute side of Aurora serverless now if I go back to the application and do a refresh you'll see that we don't get a refresh straight away there's a pause and this is because now that the database cluster experiences some incoming load it's unpausing that compute it's resuming the compute part of the cluster and this isn't an immediate process so it's important to understand that when you implement an application and use this functionality the application does need to be able to tolerate lengthier connection times now sometimes in the case of WordPress you will see an error page when you attempt to do a refresh because a timeout value within WordPress is reached before the cluster can resume in the case of this demo lesson that didn't happen it was able to resume the cluster straight away and if we go back to the RDS console and then refresh this page we'll be able to see just how many capacity units this cluster is now operating with and it's operating with two capacity units now in production usage you could be a lot more granular and customize this based on the needs of your application in my case my minimum is one and my maximum is two and my pause time frame is a relatively low five minutes because I wanted to keep it simple for this demo lesson in production usage you might have a larger range between minimum and maximum you might have a higher minimum to be able to cope with a certain level of base load and the time frame between the last access and the pausing of the compute might be significantly longer than five minutes but this demonstration lesson is just that a demo and it's just designed to highlight this at a really high level so that when it comes to you using this in production you understand the architecture now that's everything that I wanted to cover in this demo lesson it's just been a brief bit of experience of using Aurora serverless now to tidy up to return the account into the same state as it was at the start of the demo lesson just go ahead and close down all of these tabs we need to go back to the cloud formation console make sure the Aurora serverless stack is selected and then just go ahead and click on delete and then delete stack and that will remove all of those resources returning the account into the same state as it was at the start of the demo now this whole section of the course has been around trying to improve the database part of our application so we've moved from having a database running on the same server as the application we've split that off we've moved it into rds and we've evolved that from my sequel rds through to Aurora provisioned and now to Aurora serverless we still have one major limitation with our application and that's that for any posts you make on the blog the media for those posts are stored locally on the instance file system and that's something that we're going to start tackling next in the course and we're going to be using the elastic file system product or EFS at this point though that's everything that I wanted to cover in this demo lesson go ahead and complete this video and when you're ready I look forward to you joining me in the next.

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      Now the next thing that I want to demonstrate is how we can restore RDS if we have data corruption.

      The way that we're going to simulate this is to go back to our WordPress blog and we're going to corrupt part of this data.

      So we're going to change the title of this blog post from the best cats ever to not the best cats ever, which is clearly untrue.

      But we're going to change this and this is going to be our simulation of data corruption of this application.

      So go ahead and click on update to update the blog post with this new obviously incorrect data.

      Now let's assume that we need to restore this database from an earlier snapshot.

      Now let's ignore the automatic backup feature of RDS and just look at manual snapshots.

      Well, let's move back to the RDS console and click on snapshots and we'll be able to see the snapshot that we created at the start of this demo lesson.

      Remember, this does have the blog post contained within it in its original correct form.

      Now to do a restore, we need to select this snapshot, click on actions and then restore snapshot.

      Now I mentioned this in the theory lesson about backups and restores within RDS.

      Restoring a snapshot actually creates a brand new database instance.

      It doesn't restore to the existing one using normal RDS.

      So we have to restore a snapshot.

      Obviously the engine set to MySQL community and we're provided with an entry box for a brand new database identifier.

      And we're going to use a4lwordpress-restore.

      So this allows us to more easily distinguish between this and the original database instance.

      We also need to decide on the deployment option.

      So go ahead and select single DB instance.

      This is only a demo, so we don't need to select multi-AZ DB instance.

      We need to pick the type of instance that we're going to restore to.

      And again, because this is a new instance, we're not limited to the previous free tier restrictions.

      So we're able to select from any of the available instance types.

      So go ahead and select burstable classes and then pick either t2 or t3.micro.

      We'll leave storage as default.

      We'll need to provide the VPC to provision this new database instance into.

      So we'll make sure that a4l-vpc1 is selected and we'll use the same subnet group that was created by the one-click deployment, which you used at the start of this demo.

      You're allowed to choose between public access yes or no.

      We'll choose no.

      You'll have to pick a VPC security group to use for this RDS instance.

      Now the one-click deployment did create one, so click in the drop-down and select the RDS multi-AZ snap RDS security group.

      So not the instance security group, but the RDS security group.

      Once you've selected that, then delete default, scroll down.

      You can specify database authentication and encryption settings.

      And again, if applicable in the course that you're studying, I'll be covering these in a separate lesson.

      We'll leave all of that as default and click on restore DB instance.

      Now this is going to begin the process of restoring a brand new database instance from that snapshot.

      Now the important thing that you need to understand is this is a brand new instance.

      We're not restoring the snapshot to the same database instance.

      Instead, it's creating a brand new one.

      Now when this finishes restoring, when it's available for use, if we want our application to make use of it, and the restored non-corrupted data, then we're going to need to change the application to point at this newly restored database.

      So at this point, go ahead and pause the video because for the next step, which is to adjust the WordPress configuration, we need this database to be in an available state.

      So pause the video, wait for the status to change from creating all the way through to available, and then we're good to continue.

      Okay, so the snapshot restore is now completed and we have a brand new database instance, A4LWordPress-Restore.

      And in my case, it took about 10 minutes to perform that restoration.

      Now just to reiterate this concept, because it's really important, it features all the time in the exams, and you'll need this if you operate in the real world using AWS.

      If we go into the original RDS instance, just pay attention to this endpoint DNS name.

      So we have a standard part, which is the region, and then .rds, and then .amazonaws.com.

      Before this, though, we have this random part.

      Now this represents the name of the database instance as well as some random.

      If we go back to the databases list and then go into the restored version, now we can see that we have A4LWordPress-Restore.

      And this is different than that original database endpoint name for the original database.

      So the really important, the critical thing to understand is that a restore with a normal RDS will create a brand new database instance.

      It will have a brand new database endpoint DNS name, the CNAME, and you will need to update any application configuration to use this brand new database.

      So go ahead and just leave this open in this tab because we'll be needing it very shortly.

      Click on Services, find EC2, and open that in a new tab.

      So as a reminder, if we go back to the WordPress tab and just hit Refresh, we can see that we still have the corrupt data.

      Now what we want to do is point WordPress at the restored correct database.

      So to do that, go to the EC2 tab that you just opened, right click on the A4LWordPress instance, select Connect.

      We're going to use Instance Connect, so choose that to make sure the username is EC2-user and then connect to the instance.

      This process should be familiar by now because we're going to edit the WordPress configuration file.

      So cd/var/www/html, then we'll do a listing ls-la, and we want to edit the configuration file which is wp-config.php, so shudu, space, nano which is the text editor, space, wp-config.php.

      Once we're in this file, just scroll down and again we're looking for the dbhost configuration which is here.

      Now this DNS name you'll recognize is pointing at the existing database with the corrupt data.

      So we need to delete all of this just to leave the two single quotes.

      Make sure your cursor's over the second quote.

      Go back to the RDS console and we need to locate the DNS name for the A4LWordPress-Restore instance.

      Remember this is the one with the correct data.

      So copy that into your clipboard, go back to EC2 and paste that in, and then Ctrl+O and Enter to save, and Ctrl+X to exit.

      That's all of the configuration changes that we need.

      If we go back to the WordPress application and hit refresh, we'll see that it's now showing the correct post, the best cats ever, because we're now pointing at this restored database instance.

      So the key part about this demo lesson really is to understand that when you're restoring a normal RDS snapshot, you're restoring it to a brand new database instance, its own instance with its own data and its own DNS endpoint name.

      So you have to update your application configuration to point at this new database instance.

      With normal RDS, it's not possible to restore in place.

      You have to restore to a brand new database instance.

      Now this is different with a feature of Aurora which I'll be covering later in this section, but for normal RDS, you have to restore to a brand new instance.

      So those are the features which I wanted to demonstrate in this demo lesson.

      I wanted to give you a practical understanding of the types of recovery options and resilience options that you have available using the normal RDS version, so MySQL.

      Now different versions of RDS such as Microsoft SQL, PostgreSQL, Oracle, and even AWS specific versions such as Aurora and Aurora Serverless, they all have their own collections of features.

      For the exam and for most production usage, you just need to be familiar with a small subset of those.

      Generally, you'll either be using Oracle, MSSQL, or one of the open source or community versions, so you'll only have to know the feature set of a small subset of the wider RDS product.

      So I do recommend experimenting with all of the different features and depending on the course that you're taking, I will be going into much more depth on those specific features elsewhere in this section.

      For now though, that is everything that I wanted to talk about, so all that remains is for us to tidy up the infrastructure that we've used in this demo lesson.

      So go to databases.

      I want you to select the A4L WordPress -Restore instance because we're going to delete this fully.

      We're not going to be using this anymore in this section of the course, so select it, click on the Actions drop down, and then select Delete.

      Don't create a final snapshot.

      We don't need that.

      Don't retain automated backups and because we don't choose either of these, we need to acknowledge our understanding of this and type Delete Me into this box.

      So do that and then click on Delete.

      Now that's going to delete that instance as well as any snapshots created as part of that instance.

      So if we go to Snapshots, we only have the one manual snapshot.

      If we go to System Snapshots, we can see that we have one snapshot for this Restore database, and if you're deleting a database instance, then any system created snapshots for that database instance will also be deleted either immediately or after the retention period expires.

      So those will be automatically cleared up as part of this deletion process.

      We're not going to delete the manual snapshot that we created at the very start of this lesson with the catpost in because we're going to be using this elsewhere in the course.

      So leave this in place.

      Click on Databases again.

      We're going to need to wait for this Restored Database instance to finish deleting before we can continue.

      So go ahead and pause the video, wait for this to disappear from the list, and then we can continue.

      Okay, so that Restored Database instance has completed deleting.

      So now all that remains is to move back to the CloudFormation console.

      You should still have a tab open.

      Select the stack deployed as part of the one-click deployment.

      It should be called RDS Multi-AZ Snap.

      Select Delete and then confirm that deletion, and that will clear up all of the infrastructure that we've used in this demo lesson.

      It will return the account into the same state as it was at the start of this demo with one exception.

      And that one exception is the snapshot that we created of the RDS instance as part of this deployment.

      So that's everything you need to do in this demo lesson.

      I hope you've enjoyed it.

      I know it's been a fairly long one where you've been waiting a lot of the time in the demo for things to happen, but it's important for the exam and real-world usage that you get the practical experience of working with all of these different features.

      So you should leave this demo lesson with some good experience of the resilience and recovery features available as part of the normal RDS product.

      Now at this point, that's everything you need to do, so go ahead and complete this video, and when you're ready, I look forward to you joining me in the next.

    1. Welcome back and in this demo lesson we're going to continue implementing this architecture.

      So in the previous demo lesson you migrated a database from a self-managed MariaDB running on EC2 into RDS.

      In this demo lesson you're going to get the experience working with RDS's multi-availability zone mode as well as creating snapshots, restoring those snapshots and experimenting with RDS failover.

      Now in order to complete this demo lesson you're going to need some infrastructure.

      So let's move across to our AWS console.

      You need to be logged in to the general AWS account.

      So that's the management account of the organization and as always make sure that you have the Northern Virginia region selected.

      Now attached to this lesson is a one-click deployment link so go ahead and open that.

      This will take you to a quick create stack page and everything should be pre-populated and ready to go.

      So the stack name is RDS multi-AZ snap.

      All of the parameters have default values.

      Multi-AZ is currently set to false so leave that at false, check the capabilities box at the bottom and then click on create stack.

      Now this infrastructure will take about 15 minutes to apply and we need it to be in a create complete state before we continue.

      So go ahead, pause the video and resume it once CloudFormation has moved into a create complete state.

      Okay so now that this stack has moved into a create complete state we need to complete the installation of WordPress and add our test blog post because we're going to be using those throughout this demo lesson.

      Now this is something that you've done a number of times before so we can speed through this.

      So click on the services drop down, move to the EC2 console.

      We need to go to running instances and we'll need the public IP version 4 address of A4L-WordPress.

      So go ahead and copy the public IP version 4 address into your clipboard.

      Don't use this open address.

      Open that in a new tab.

      We'll be calling the site as always the best cats for username put admin for the password.

      We'll be using the animals for life strong password and then as always test at test.com for the email address.

      Enter all of that and click on install WordPress.

      Then you'll need to log in admin for the username and to the password click on login.

      Once we logged in go to posts click on trash under hello world to delete the existing post and then add a new post.

      Close down this dialogue for the title of the post the best cats ever click on the plus select gallery.

      At this point go ahead and click the link that's attached to this lesson to download the blog images.

      Once downloaded extract that zip file and you'll get four images.

      Once you've got those images ready click on upload locate those images select them and click on open and that will add those to the post.

      Once they're fully loaded in we can go ahead and click on publish and then publish again and that will publish this post to our blog.

      And as a reminder that stores these images on the local instance file system and adds the post metadata to the database and that's now running within RDS.

      Now I want to step through a few pieces of functionality of RDS and I want you for a second to imagine that this blog post is actually a production enterprise application.

      Maybe a content management system and I want to view all of the actions that we perform in this demo lesson through the lens of this being a production application.

      So go ahead and return to the AWS console click on services and we're going to move back to RDS.

      The first thing that we're going to do is to take a snapshot of this RDS instance.

      So just close down any additional dialogues that you see go to databases.

      Then I want you to select the database that's been created by the one click deployment link that you used at the start of this demo lesson.

      Then select actions and then we're going to take a snapshot.

      Now a snapshot is a point in time copy of the database.

      When you first do a snapshot it takes a full copy of that database so it consumes all of the capacity of the data that's being used by the RDS instance.

      So this initial snapshot is a full snapshot containing all of the data within that database instance.

      Now we're going to take a snapshot and we're going to call it a four L wordpress hyphen with hyphen cat hyphen post hyphen mySQL hyphen and then the version number without any dots or spaces.

      Now depending on when you're watching this video doing this lesson you might have been using a different version of SQL.

      And so in the lesson description for this lesson I've included the name of the snapshot that you need to use.

      So go ahead and check that now and include that in this box.

      So that informs us what it is, what it contains and the version number that this snapshot refers to.

      So go ahead and enter that and then click on take snapshot and that's going to begin the process of creating this snapshot.

      Now the process takes a variable amount of time.

      It depends on the speed of AWS on that particular day.

      It depends on the amount of data contained within the database and it also depends on whether this is the first snapshot or a subsequent snapshot.

      Now the way that snapshots work within AWS is the first snapshot contains a full copy of all of the data of the thing being snapshotted and any subsequent snapshot only contains the blocks of data which have changed from that last previous successful snapshot.

      So of course the first snapshot always takes the longest and everything else only takes the amount of time required to copy the changed data.

      So if we just give this a few minutes let's keep refreshing.

      Mine's still reporting at 0% complete so we need to allow this to complete before we move on.

      So go ahead and pause the video and resume it once your snapshot has completed.

      And there we go our snapshots now moved into an available status and the progress has completed.

      And in my case that took about five minutes to complete from start to finish.

      So again just to reiterate this snapshot has been taken.

      It's a copy of an RDS MySQL database of a particular version and it contains our WordPress database together with the cat post that we just added.

      And that's important to keep in mind as we move on with the demo lesson.

      Now you could go ahead and take another snapshot and this one would be much quicker to complete.

      It would only contain any data changed between the point that you take it and when you took this previous snapshot.

      I'm not going to demonstrate that in this video but you can do that.

      And for production usage you may use snapshots in addition to the normal automated backups provided by RDS.

      Snapshots that you take manually live past the life cycle of the RDS instance.

      And if you want to tidy them up you have to do that manually or by using scripts that you create.

      So snapshots that are taken manually are not managed by RDS in any way.

      And that's important to understand from a DR and the cost management perspective.

      Now the next thing that I want to demonstrate is the multi AZ mode of RDS.

      So if we go back to the RDS console just expand this menu and go to databases.

      Currently this database is using a single RDS instance.

      So this RDS instance is not resilient to the failure of an availability zone within this region.

      Now to change that process we can provision a standby replica in another availability zone and that's known as multi AZ.

      Now it's worth noting that this is not included within the AWS free tier.

      So there will be a small charge to do this optional step to enable multi AZ mode.

      Make sure that you have the database instance selected and then click on modify.

      Now it's on this screen that we can change a lot of the options which relate to this entire RDS instance.

      We've got the option to adjust the database identifier, provide a new database admin password.

      We can change the DB instance size or type if we want.

      We can adjust the amount of storage available to the database instance, even enable storage auto scaling.

      But what we're looking for specifically is adjusting the availability and durability settings.

      Currently this is set to do not create a standby instance and we're going to modify this.

      We're going to change it to create a standby instance and this is something that's recommended for any production usage.

      This creates a standby replica in a different availability zone.

      So it picks another availability zone, specifically another subnet that's available within the database subnet group that was created by the one click deployment.

      So we're going to set that option and scroll down and then select continue.

      Now because we have a maintenance window defined on this RDS instance, we have two different options of when to apply this change.

      We can either apply the change during the next scheduled maintenance window.

      Remember, this is a definable value that you can set when you create an RDS instance or you modify its settings.

      Or we can specify that we want to apply immediately the change that we're making.

      And for this demo lesson, that's what we're going to do.

      Now it does warn you that any changes could cause a performance impact and even an outage.

      So it's really important that if you are applying changes immediately, you understand the impact of those changes.

      So make sure that you have apply immediately selected and then click on modify DB instance.

      Now a multi AZ deployment is essentially an automatic standby replica in a separate availability zone.

      What happens behind the scenes is that the primary database instance is synchronously replicated into this standby replica inside a different availability zone.

      Now this provides a few benefits.

      It provides data redundancy benefits.

      It means that any operations which can interrupt IO such as system backups will occur from the standby replica.

      So won't impact the primary database and that provides a real advantage for production RDS deployments.

      But the main reason beyond performance is that it helps protect any databases in the primary instance against failure of an availability zone.

      So if the availability zone of the primary instance fails and then the C name of the database will be changed to point at the standby replica.

      And that will minimize any disruption to your application and its users.

      Now if we just hit refresh, we can see the status is modifying and what's happening behind the scenes is AWS are taking a snapshot of the primary DB instance.

      It's restoring that snapshot into the standby replica, which is in a different availability zone.

      And then it's setting up synchronous replication between the primary and the standby replica.

      So this is a process which happens behind the scenes.

      But it does mean that we need to wait for this process to be complete until the process is complete.

      This is not a multi AZ deployment.

      So go ahead and pause the video and wait for the status to change away from modifying.

      We need this to be in an available state in order to continue with the demo.

      So go ahead and pause the video and resume it once this modification has completed.

      Okay, so the status has now changed to available.

      And in my case, it took about 10 minutes to enable multi AZ mode.

      So that's the provisioning of a standby replica in another availability zone.

      Now, the likelihood of an AZ failure happening while I'm recording this demo lesson is relatively small, but we can simulate a failure to do that.

      If we have the database instance selected and then select the actions drop down and then reboot, we can use the option reboot with failover.

      If we choose this option, then part of the process is that a simulated failover occurs.

      So the C name, the database endpoint, that's moved so that it now points at the standby replica and then the old primary instance is restarted.

      So that's what we're going to do to simulate this process.

      So go ahead and select to reboot the database instance.

      Make sure that you have reboot with failover selected and then click on confirm.

      And this will begin the process of rebooting the database instance.

      Now, if we go back to the WordPress blog and we click on view post, you'll see that right away it's not immediately loading.

      And that's because the failover from the primary to the standby isn't immediate.

      Failover times are typically 60 to 120 seconds.

      So that's important to keep in mind if you're deploying RDS in a business critical situation.

      It doesn't offer immediate failover.

      So let's just stop this and hit reload again.

      And now we can see that the page is starting to load because the C name for the database has been moved from pointing at the primary to pointing at the standby replica, which is the new primary.

      Okay, so this is the end of part one of this lesson.

      It was getting a little bit on the long side and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part two will be continuing immediately from the end of part one.

      So go ahead, complete the video and when you're ready, join me in part two.

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      Okay, so the instance is now in an available state.

      Let's just close down this informational dialogue at the top.

      And let's just minimize this menu on the left.

      Let's maximize the amount of screen space that we have for this specific purpose.

      So I just want us to go inside this database instance and explore together the information that we have available.

      So I talked in the theory lesson how every RDS instance is given an endpoint name and an endpoint port.

      So this is the information that we'll use to connect to this RDS instance.

      Networking wise, this instance has been provisioned in US-EAST-1A.

      It's in the Animals for Life VPC and it's used our A4L subnet group that we created at the start of this demo.

      And that means that it's currently utilizing all three database subnets in the Animals for Life VPC.

      But it's chosen because we only deployed one instance to use US-EAST-1A.

      Now this is the VPC security group that we're going to need to configure.

      So right click on this and open it in a new tab and move to that tab.

      This is the security group which controls access to this RDS instance.

      So let's expand this at the bottom.

      So currently it has my IP address being the only source allowed to connect into this RDS instance.

      So the only inbound rule on the security group protecting this RDS instance is allowing my IP address.

      So we're going to click on Edit and then click on Add Rule.

      And we're going to add a rule which allows our other instances to connect to this RDS instance.

      So first in the type drop down click and then type mySQL to get the same option as the line above and then click to select.

      Next go ahead and type instance into the source box and then select the migrate to RDS-instance security group.

      Now this is the security group that's used by any instances deployed by our one click deployment.

      And this allows those instances to connect to our RDS instance and that's what we want.

      So go ahead and select that and then click on Save Rules.

      And this means now that our WordPress instance can communicate with RDS.

      So now let's move back to the RDS tab and then make sure we're inside the A4L WordPress database instance.

      So that's the connectivity and the security tab.

      We also have the monitoring tab and it's here where you can see various CloudWatch provided metrics about the database instance.

      You also have logs and events related to this instance.

      So if we go and have a look at recent events we can see all of the events such as when the database instance was created, when its first backup was created.

      And you can explore those because they might be different in your environment.

      You can click on the Configuration tab and see the current configuration of the RDS instance.

      The Maintenance and Backups tab is where you can configure the maintenance and backup processes and then of course you can tag the RDS instance.

      Now in other lessons in this section of the course and depending on what course you're taking I will be talking about many of these options, what you can modify and which actions you can perform on RDS instances.

      But for now we're just going to move on with this demo.

      So the next step is that we need to migrate our existing data into this RDS instance.

      So what we're going to do is to click on the Connectivity and Security tab and we're going to leave this open.

      We're going to need this endpoint name and port very shortly.

      You should still have a tab open to the EC2 console.

      If you don't you can reach that by going on Services and then opening EC2 in a new tab.

      But I want you to select the A4L-WordPress instance and then right click and connect to it using Instance Connect.

      So go ahead and do that.

      Once you've done that we're going to start referring to the lesson commands document.

      So make sure you've got that open.

      We're going to use this command to take a backup of the existing MariaDB database.

      So we need to replace a placeholder.

      What we need to do is delete this and replace it with the private IP address of the MariaDB EC2 instance.

      So go back to the EC2 console, select the DB-WordPress instance and copy the private IP version 4 address into your clipboard.

      And then let's move back to the WordPress instance and paste that in.

      Go ahead and press Enter and it will prompt you for the password.

      And the password is the same Animals for Life strong password that we've been using everywhere.

      Copy that into your clipboard.

      So this is the password for the A4L WordPress user on the MariaDB EC2 instance.

      So paste that in and press Enter and then LS-LA to confirm that we now have this A4L WordPress.SQL database backup file.

      And we do, so that's good.

      So as we did in the previous demo lesson, we're going to take this backup file and we're going to import it into the new destination database, which is going to be the RDS instance.

      To do that, we'll use this command, but we're going to need to replace the placeholder hostname with the CNAME of the RDS instance.

      So go ahead and delete this placeholder, then go back to the RDS console and I'll want you to copy the endpoint name into your clipboard.

      So select it, right click and then copy.

      We won't need the port number because this is the standard MySQL port and if you don't specify it, most applications will assume this default.

      So just make sure that you have the endpoint DNS name or endpoint CNAME in your clipboard.

      And then back on the WordPress EC2 instance, go ahead and paste this database name into this command and press Enter.

      And again, you'll be asked for the password and that's the same Animals for Life strong password.

      So copy that into your clipboard, paste that in and press Enter.

      And that's imported this A4LWordPress.SQL file into the RDS instance.

      So now we need to follow the same process and change WordPress so that it points at the RDS instance.

      And we do that by moving to where the WordPress configuration file is.

      So cd space forward slash var forward slash ww w forward slash html and press Enter.

      And then shudu.

      So we have admin privileges, nano, which is a text editor and then wp-config.php and press Enter.

      Then we need to scroll down and we're looking for where it says DB host and currently it has a host name here.

      Now if you go back to the EC2 console and you look at the A4L-DB-WordPress instance, you'll see that its private IP version for DNS name is what's listed inside this configuration item.

      So it's currently pointing at this dedicated database instance.

      What we need to do is replace that and we're going to replace it with the RDS database DNS name or the CNAME of this RDS instance.

      So copy that into your clipboard and then go ahead and delete this private DNS name for the MariaDB EC2 instance and then paste in the RDS endpoint name, also known as the RDS CNAME.

      Once you've done that, control O and Enter to save and control X to exit.

      And now our WordPress instance is pointing at the RDS instance for its database.

      Now we can verify that by checking WordPress, move back to instances, select the WordPress instance, copy the public IP version for addressing to your clipboard.

      Don't use this open address link.

      Open that in a new tab.

      Go ahead and just click on the best cats ever to verify the functionality and it does look as though it's working.

      And to verify that, if we go back to the EC2 console, select the A4L-DB-WordPress instance and right click and then stop that instance.

      Now the original database that was providing database services to WordPress is going to move into a stopped state.

      And if our WordPress blog continues functioning, we know that it's using the RDS instance.

      So let's keep refreshing and wait for this to change into a stopped state.

      There we go.

      It's stopped.

      And if we go back to our WordPress page and refresh, it still loads.

      And so we know that it's now using RDS for its database services.

      So at this point, that's everything that I wanted you to do in this demo lesson.

      You've stepped through the process of provisioning an RDS instance.

      So you've created a subnet group, provisioned the instance itself, explored the functionality of the instance, including how to provide access to it by selecting a security group.

      And then editing that security group to allow access.

      You've performed a database migration and you've explored how the RDS instance is presented in the console.

      So that's everything that you need to do within this demo lesson.

      And don't worry, we're going to be exploring much more of the advanced functionality of RDS as we move through this section of the course.

      For now, though, I want us to clear up the infrastructure that we've created as part of this demo lesson.

      Now, because we've provisioned RDS manually outside of CloudFormation, unfortunately, there is a little bit more manual work involved in the cleanup.

      So I want you to go to the RDS console, move to databases, select this database, click on actions, and then select delete.

      Now it will prompt you to create a final snapshot and we're not going to do that.

      We're not going to retain automated backups and so you'll need to acknowledge that upon instance deletion, automated backups including any system snapshots and pointing time recoveries will no longer be available.

      And don't worry, I'll be talking about backups and recovery in another lesson in this section of the course.

      For now, just acknowledge that and then type delete me into this box and confirm the deletion.

      Now this deletion is going to take a few minutes.

      It's not an immediate process.

      It will start in a deleting state and we need to wait for this process to be completed before we continue the cleanup.

      So go ahead and pause this video and wait for this instance to fully delete before continuing.

      Now that the instance has been deleted, it vanishes from this list.

      Next, we need to delete the subnet group that we created earlier.

      So click on subnet groups, select the subnet group and then delete it.

      You'll need to confirm that deletion.

      Once done, it too should vanish from that list.

      Next, go to the tab you've got open to the VPC console, scroll down and select security groups.

      Now look through this list and locate the security group that you created as part of provisioning the RDS instance.

      It should be called a4LVPC-RDS-SG.

      Select that, click on actions and then delete security group and you'll need to confirm that process as well.

      Once that's deleted, the final step is to go to the cloud formation console and then you'll need to delete the cloud formation stack that was created using the one-click deployment at the start of the demo.

      It should be called migrate to RDS.

      Select it, click on delete and confirm that deletion.

      And once deleted, the account will be returned into the same state as it was at the start of the demo lesson.

      So all of the infrastructure that we've used will be removed from the account and the account will be in the same state as at the start of the demo.

      Now I hope you've enjoyed this demo and that we're repeating the same WordPress installation and then the creation of the blog post over and over again.

      But I want you to get used to different parts of this process over and over again.

      You need to know why not to use a database on EC2.

      You need to know why not to perform a lot of these processes manually.

      From this point onward in the course, we're going to be using RDS to evolve our WordPress design into something that is truly elastic.

      And so all of these processes, the things I'm having you repeat are really useful to aid in your understanding of all of these different components.

      So from this point onward, we're going to be automating the creation of RDS and focusing on the specific pieces of functionality that you need to understand.

      But at this point, that's everything that you need to do in this demo.

      So go ahead, complete the video and when you're ready, I look forward to you joining me in the next.

    1. Welcome back and in this demo lesson you're going to get some experience of how to provision an RDS instance and how to migrate a database from an existing self-managed MariaDB database instance through to RDS.

      So over the next few demo lessons in this section of the course, you're going to be evolving your database architecture.

      We're going to start with a single database instance, then we're going to add multi-AZ capability as well as talking about backups and restores.

      But in this demo lesson specifically, we're going to focus on provisioning an RDS instance and migrating data into it.

      Now in order to get started with this demo lesson, as always make sure that you're logged into the general AWS account, so the management account of the organization and you need to have the Northern Virginia region selected.

      Now attached to this lesson is a one-click deployment link that you'll need to use to provision this demo lesson's infrastructure.

      So go ahead and click on that link now.

      That's going to move you to a quick create stack screen.

      The stack name should be pre-populated with migrate to RDS.

      Scrolling down all of the parameter values will be pre-populated.

      All you need to do is to click on the capabilities checkbox and then create stack.

      There's also a lesson commands document linked to this lesson and I'd suggest you go ahead and open that in a new tab because you'll be referencing it as you move through this demo lesson.

      Now you'll notice that this will look similar to the previous demo lesson's lesson commands document, but it has one small difference.

      The initial command way of doing the backup of the source database, because that source database is going to be stored on a separate MariaDB database running on a separate EC2 instance, instead of taking the backup from the local instance, in this case it's connecting to a separate EC2 instance.

      Otherwise, most of these commands are similar to the ones you used in the previous demo lesson.

      Now you're going to need to wait for this stack to move into a create complete state before you continue the demo.

      So go ahead and pause the video, wait for your stack to change to create complete and then you're good to continue.

      Okay, so that cloud formation stack has now moved into a create complete state and it's created a familiar set of infrastructure.

      Let's go ahead and click on the services drop down and then move to the EC2 console and just take a look.

      So if we click on instances, you'll see that we have the same two instances as you saw in the previous demo lesson.

      So we have A4L-WordPress, which is running the Apache web server and the WordPress application.

      And then we have A4L-DB-WordPress and this is running the separate MariaDB database instance.

      So what we need to do in order to perform this migration is first create the WordPress blog itself and the sample blog post.

      And this is the same thing that we did in the previous demo.

      So we should be able to go through this pretty quickly.

      So go ahead and select the A4L-WordPress instance and copy its public IP version for address into your clipboard and then open that in a new tab.

      And again, make sure not to use the open address because this uses HTTPS.

      So copy the public IP version for address and then open that in a new tab.

      Again, we're going to call the site the best cats.

      We're going to use admin for the username.

      And then for the password, let's go back to the CloudFormation tab.

      Make sure you've got the migrate to RDS stack selected and then click on parameters.

      We're going to use the same database password.

      So copy that into your clipboard and replace the automatically generated one with the animals for live complex password.

      And then enter test@test.com into the email box and click on install WordPress.

      Once installed, click on login.

      You'll need to use the admin username and the same password.

      Click on login.

      Then we're going to go to posts.

      We're going to select the existing Hello World post.

      Select trash this time.

      Then click on add new.

      Close down this dialog for title.

      We're going to use the best cats ever.

      Click on the plus.

      Select gallery.

      At this point, go ahead and click the link that's attached to this lesson to download the blog images.

      Once downloaded, extract that zip file and you'll get four images.

      Once you've got those images ready, click on upload, locate those images, select them and click on open.

      Wait for them to load in.

      Select publish and publish again.

      And that saved the images onto the application instance and added the data for this post onto the separate MariaDB database.

      So now we have this simple working blog.

      Let's go ahead and look at how we can provision an RDS instance and how we can migrate the data into that RDS instance.

      So move back to the AWS console.

      Click on the services drop down and type RDS into the search box and open that in a new tab.

      Now, as I've mentioned in the theory parts of this section, RDS is a managed database server as a service product from AWS.

      It allows you to create database instances and those instances can contain databases that your applications can make use of.

      Now to provision an RDS instance, the first thing that we need to do is to create a subnet group.

      Now a subnet group is how we inform RDS which subnets within a VPC we want to use for our database instance.

      So first we need to create a subnet group.

      So select subnet groups on the menu on the left and then create a DB subnet group.

      Now we're going to use a4lsn group, so animals for life subnet group for both the name and for the description.

      And then select the VPC drop down and we're going to select the a4l-vpc1vpc.

      So this is the animals for life VPC which has been created by the one click deployment that you used at the start of this demo.

      Now once we've selected a name and a description and a VPC for this subnet group, then what we need to do is select the subnets that this database will be going into.

      So we're going to select the database subnets in US East 1A, US East 1B and US East 1C.

      So click on the availability zone drop down and pick those three availability zones.

      So 1A, 1B and 1C.

      Once we've selected the availability zones that this subnet group is going to use, next we pick the subnets.

      So click on the drop down.

      Now we want to pick the database subnets within the animals for life VPC and all we can see here are the IP address ranges.

      So to help us with this click on the services drop down, type VPC and then open that in another new tab.

      Once that loads, go ahead and click on subnets, sort the subnets by name and then locate sn-dba, dbb and dbc.

      And just move your cursor across to the right hand side and note what the IP address ranges are for those different database subnets.

      So 16, 80 and 144.

      Go back to the RDS console, click on the subnets drop down and we need to pick each of those three subnets.

      So 16, 80 and 144.

      So these represent the database subnets in availability zone 1A, 1B and 1C.

      And then once we've configured all of that information, we can go ahead and click on create to create this subnet group.

      So this subnet group is something that we use when we're provisioning an RDS instance.

      And as I mentioned moments ago, it's how RDS determines which subnets to place database instances into.

      Now when we're only using a single database instance, then that decision is fairly easy.

      But RDS deployments can scale up to use multiple replicas in multiple different availability zones.

      You can have multi-AZ instances, read replicas.

      Aurora has a cluster architecture which we'll talk about later in this section.

      And so subnet groups are essential to inform RDS which subnets to place things into.

      So now that we've configured that subnet group, let's go ahead and provision our RDS database instance.

      So to do that, click on databases and then we're going to create a database.

      So click on create database.

      Now when you're creating a database, you have the option of using standard create where you have visibility of all of the different options and then easy create which applies some best practice configurations.

      Now I want you to get the maximum experience possible, so we're going to use standard create.

      Now when you're creating an RDS database instance, you have the ability to pick from many different engines.

      So some of these are commercial like Oracle or Microsoft SQL Server.

      And with some of these, you have the option of either paying for a license included with RDS or you can bring your own license.

      For other database engines, there isn't a commercial price to pay for their usage and so they're much cheaper to use.

      But you should select the engine type which is compatible with your application.

      Now we're going to be talking about Amazon Aurora in dedicated lessons later in this section of the course.

      Amazon Aurora is an AWS designed database product which has compatibility with MySQL and PostgreSQL.

      For this demo lesson, we're going to use MySQL.

      So go ahead and select MySQL and it's going to be using MySQL Community Edition.

      So now let's just scroll down and step through some of the other options that we get to select when provisioning an RDS instance.

      Now for all of these database engines, you have the ability to pick different versions of that engine.

      And this is fairly critical because there are different major and minor versions that you can select from.

      And different versions of these have different limitations.

      So for example, we're going to be talking about snapshots later in this section.

      And if you want to take a snapshot of an RDS database and then import that into an Aurora cluster, you need to pick a compatible version.

      And then Aurora Serverless which we'll be talking about later on in this section has even more restrictions.

      Now to keep things simple, I want you to ignore what version I pick in this video and instead look in this lesson's description and pick the version that I indicate in the lesson description because I'll keep this updated if AWS make any changes.

      Now you can choose to use a template.

      These templates give you access to only the options which are relevant for the type of deployment that you're trying to use.

      So in production, you would pick the production template.

      If you have any smaller or less critical dev or test workloads, then you could pick this template.

      If you want to ensure that you can only select free tier options, then you should pick this template.

      And that's what we're going to do in this demo because we want this demo to fall under the free tier.

      So click on the free tier template.

      I'll be talking about availability and durability later in this section because we've selected free tier only.

      We don't have the ability to create a multi AZ RDS deployment.

      And now we need to provide some configuration information about the database instance specifically.

      So the first thing that we need to do is to provide a database instance identifier.

      So this is the way that you can identify one particular instance from any other instances in the AWS account in the current region.

      So this needs to be unique.

      So we're going to use a four L WordPress for this database instance.

      Then we need to pick a username which will be given admin privileges on this database instance.

      And we're going to replace admin with a four L WordPress.

      So we're going to use this for both the database identifier and the admin user of this database.

      Now for the password for this admin user, we're going to move back to the cloud formation console and we're going to use this same animals for life complex password.

      So copy that into your clipboard and paste it in for the password and the confirm password box.

      And this just keeps things consistent between the self banished database and the RDS database.

      Scroll down further still and it's here where you can select the database instance class to use.

      Now because we've selected free tier only, we're limited as to what database size and type we can pick.

      If we'd have selected production or dev test from the templates above, we would have access to a much wider range of database instance classes, both standard, memory optimized and burstable.

      But because we've selected the free tier template, we're limited as to what we can select.

      Now this might change depending on when you're watching this demonstration, but at the point I'm recording this video, it's db.t3.micro.

      So don't be concerned if you see something different in this box.

      Just make sure that you select the type of instance which falls under the free tier.

      Then continue scrolling down and we need to pick the size of storage and the type of storage to use for this RDS instance.

      Now whether you need to select this is dependent on what engine type you pick.

      If you select Aurora, which we'll be talking about later on in this section, then you don't need to pre-allocate storage.

      If you're using the MySQL version of RDS, then you do need to set a type of storage and a size of storage.

      Now we're going to use the minimum which is 20GIB because our requirements for this database are relatively small.

      And if we wanted to, if this was production, we could set storage autoscaling.

      And this allows RDS to automatically increase the storage when a particular threshold is met.

      But again, because this is a demo and it's only using a very small blog, we don't need storage autoscaling.

      So go ahead and uncheck that option.

      Now we need to select a VPC for this RDS instance to go into.

      So click in the drop down and select the Animals for Life VPC.

      So that's A4L-VPC1.

      And then we need to pick a subnet group.

      Now this is the thing that we've just created.

      We only have one in this account, so there's nothing else to select.

      But this is how we can advise RDS on which subnets to use inside the VPC.

      Scroll down further still and we can specify whether we want this database to be available.

      We want this database to be publicly accessible.

      So this is whether we want instances and devices outside the VPC to be able to connect to this database.

      This obviously comes with some security trade-offs.

      And because we don't need that in this demonstration, because the only thing that we want to connect to this RDS instance is our WordPress instance, which is in the same VPC, then we can select Not to Use Public Access.

      So make sure the No option is selected.

      Now the way that you control access to RDS is you allocate a VPC security group to that instance.

      So we could either choose an existing security group or we could create a new one.

      So it's this security group which surrounds the network interfaces of the database and controls access to what can go into that database.

      So we want to create a new VPC security group.

      So we want to make that option.

      We're going to call the security group A4LVPC-RDS-SG.

      And we need to remember to update this so that our WordPress instance can communicate with our RDS instance.

      And we'll do that in the next step.

      If we wanted to pick a specific availability zone for this instance to go into, then we could select one here or we can leave it up to RDS to pick the most suitable.

      So we can select No Preference.

      Continue scrolling down.

      We won't change the Database Authentication option because we want to allow password authentication.

      Continue scrolling down and we're going to expand Additional Configuration.

      By default, an RDS instance is created with no database on that instance.

      In this case, because we're migrating an existing WordPress database into RDS, we're going to go ahead and create an initial database.

      And to keep things easy and consistent, we're going to use the same name, so A4L WordPress.

      Now you can enable automatic backups for RDS instances.

      And I'll be talking about these in a separate theory lesson.

      If you do select automatic backups, then you can also pick a backup retention period as well as a backup window.

      So we've got Advanced Monitoring, various log exports.

      We don't need to use any of those.

      You can also set the Maintenance window for an RDS instance.

      So when Maintenance will be performed, you can enable Deletion Protection if you want.

      If this is a production database, we don't need to do that.

      What we're going to do is scroll all the way down to the bottom and then click on Create Database.

      Now this process can take some time.

      I've seen it take anywhere from five to 45 minutes.

      And we're going to need this to be finished before we move on to the next step.

      So this seems like a great time to end this video.

      It gives you the opportunity to grab a coffee or stretch your legs.

      Wait for this database creation to finish.

      And then when you're ready, I'll look forward to you joining me in part two of this video.

    1. Welcome to this demo lesson where you're going to migrate from the monolithic architecture on the left of your screen towards a tiered architecture on the right.

      Essentially you're going to split the WordPress application architecture, you're going to move the database from being on the same server as the application to being on a different server and this will form step one of moving this architecture from being a monolith through to being a fully elastic architecture.

      Now this is the first stage of many but it is a necessary one.

      Now in order to perform this demonstration you're going to need some infrastructure.

      Before we apply the infrastructure just make sure that you're logged in to the general AWS account, so the management account of the organization and as always you need to have the Northern Virginia region selected.

      Now once you've got both of those set there's a one-click deployment link attached to this lesson so go ahead and click on that link.

      What this is going to do is deploy the Animals for Life base infrastructure, it's going to deploy the monolithic WordPress application instance and it's also going to deploy a separate MariaDB database instance that you're going to use as part of the migration.

      Now everything set, the stack name should be set to a suitable default, all you need to do is to scroll all the way down to the bottom, check this capabilities box and click on create stack.

      Now also attached to this lesson is a lesson commands document which contains all the commands you'll be using throughout this demo.

      So go ahead and open that in a new tab, you'll be referencing it constantly as you're making the adjustments to the WordPress architecture.

      Now we're going to need this CloudFormation stack to be fully complete before we can continue so go ahead and pause the video and resume once the CloudFormation stack moves into a create complete state.

      So now the stacks moved into a create complete state, we're good to continue.

      Now this has created the base Animals for Life infrastructure which includes a number of EC2 instances so let's take a look at those, let's click on services and then locate and open EC2 in a brand new tab.

      Once you're at the EC2 console if you do see any dialogues around user interface updates then just go ahead and close those down and then click on instances running.

      Once you're here you'll see two EC2 instances, one will be called A4L-WordPress and this is the monolith so this is the EC2 instance which contains the WordPress application and the built-in database.

      So this is the WordPress installation that we're going to migrate from and then this instance A4L-DB-WordPress this contains a standalone MariaDB installation so we're going to migrate the database for WordPress from this instance onto the DB instance and this will create a tiered application architecture rather than the monolith which we currently have.

      So step number one is to perform the WordPress installation so to do that I want you to go ahead and copy the public IP version for address of the WordPress EC2 instance into your clipboard and then open it in a new tab.

      Now be careful not to use the open address link that will use HTTPS which we're not currently using so copy the IP address into your clipboard and open that in a new tab.

      Now when you do that you'll see a familiar WordPress installation dialog we're going to create a simple blog for site title go ahead and call it the best cats for username pick admin and then for the password instead of using the randomly selected one go ahead and use this same complex password that we've used for the CloudFormation template so this is animals for life but with number substitution.

      So if you go back to your CloudFormation tab and go to the parameters tab this is the same password that we use for the DB password and the DB root password.

      Now of course in production this is incredibly bad practice we're just doing it in this demo to keep things simple and avoid any mistakes.

      So back to the WordPress installation screen site title the best cats username admin this for the password and then just go ahead and type a fake email so I don't want to use my real email for this I'm going to type test at test.com you can do the same and then go ahead and click on install WordPress so this is installed the WordPress application and it's using the Maria DB server that's on the same EC2 instance so part of the same monolith.

      So we're going to log in we'll need to type admin and then use the animals for life strong password and click on login and once we logged in we're going to create a simple blog post so click on posts we're going to select the existing hello world post select trash this time then click on add new then we're going to add a new post we can close down this introduction dialogue and for title go ahead and type the best cats ever and then some exclamation points next click on this plus sign and we're going to add a gallery now at this point you're going to need some images to upload to this blog post I've attached an images link to this lesson so if you go ahead and click that link it will download a zip file if you extract that zip file it's going to contain four image files all four of my cats so at this point once you've downloaded and extracted that file go ahead and click on upload locate those images there should be four select them all and click on open that will add these images to this blog post and once you've added them all you can go ahead and click on publish and then publish again and this will publish this blog post so it will add data to the database that's running on the monolithic application instance as well as store these images on the local instance file system now making a point of mentioning that these images are stored on the file system because as you'll see later in the course this is one of the things that we need to migrate when we're moving to a fully elastic architecture we can't have images stored on the instances themselves we need to move that to a shared file system for now though we're focusing on the database so at this point we have the working blog the images for this blog are stored on the local file system of a4l-wordpress and the data for that blog post is stored on the MariaDB database that's also running on this EC2 instance so the next step of this demo lesson is that you're going to migrate the data from a4l-wordpress onto a4l-db-wordpress and this is an isolated MariaDB instance this is dedicated for the database so to do this migration select a4l-wordpress right click we're going to connect to this instance we'll be using EC2 instance connect so just make sure that the username is set to EC2-user and then click on connect now this is where you're going to be using the commands that are stored within the lesson commands document so you need to make sure that you have this ready to reference because it's far easier to copy and paste these commands and then adjust any placeholders rather than type them out manually because that's prone to errors the first step is to get the data from the database that's running on this monolithic application instance and store it in a file on disk so that's the first thing we need to do we need to do a backup of the database into a .sql file now to do that we use this command so it's a utility called my sql-dump it uses the -u to specify the user that we're going to be using to connect to the database then we use -p to specify that we want to provide a password and we could either provide the password on the command line or we could have it prompt us now if we supply the password with no spaces next to this -p then it will accept it as input on this command if we don't specify anything so there's a space here then it's going to ask us for the password the next thing we specify is the database name that we want to do the dump of in this case it's a4l WordPress which is the database for the animals for life WordPress instance now if we just run this command on its own it would output the dump so all of the data in the database to standard output which in this case is our screen we don't want it to do that we want it to store the results in a file called a4l WordPress dot sql and so we use this symbol which means that it's going to take the output of this component of the command and it's going to redirect it into this file so let's go ahead and run this command and it's going to prompt us for the password for this database now to get that go back to cloud for information make sure parameters are selected and it's this password that we need which is the DB password so copy that into your clipboard go back to the instance paste that in press enter and that will output all the data in the database to this file now you won't see any indication of success or failure but if you do an LS space -la and press enter one of the files that you'll see is a4l WordPress dot sql so now we have a copy of the WordPress database containing our blog post the next thing that we need to do is to take this file this backup of the database and inject it into the new database that we want to use so the dedicated Maria DB EC2 instance and to do that we're going to use this command so this command has two components the first component is this which connects to the Maria DB database instance the second component is this which takes the backup that we've just made and it feeds it into this command so this backup contains all the necessary definitions to create a new database and inject the data required this component of the command just allows us to connect to this new dedicated Maria DB instance now there are some place holders that we need to change the database name that we're going to use is the same so a4l WordPress we're still going to want to be prompted for a password so -p is what we use this time though we're going to connect using a user called a4l WordPress so we're not using the root user we're going to connect to this separate Maria DB database instance using a user a4l WordPress the only thing that we need to change is that we need to connect to a non-local host so when we used the mysql dump command we didn't specify a host to connect to and this defaulted to local host so the current machine in the case of this command we're operating with a separate server this dedicated EC2 instance which is running the Maria DB database server so a4l -db -wordpress we need to connect to this so what we'll need to connect to this is the private IP version for address of this separate database instance so select it look for private IP version for addresses and then click on the icon next to this to copy the private IP version for address of this separate database server into your clipboard then return to the application instance and we need to replace the placeholder here with that value so make sure that you're one space after the end of this placeholder and just delete this leave a space between -h and where the cursor is and then paste in that IP address so this is going to connect to this separate EC2 instance using its private IP it's going to use the a4l WordPress user it will prompt us for a password it will perform the operation on the a4l WordPress database and it's going to use the contents of this backup file to perform those tasks so go ahead and press enter and you'll be prompted for a password now again it's the same password this has all been set up as part of the cloud formation one-click deployment this lesson is about the migration process not setting up a database server so I've automated this component of the infrastructure so copy the DB password into your clipboard go back to the instance paste it in and press enter so now we've uploaded our WordPress application database into this separate MariaDB database server the next step is to configure WordPress to point at this new database server so to do that cd space forward slash var forward slash www forward slash html and press enter and then we're going to run a shudu space nano which is a text editor space wp-config.php and this is the WordPress configuration file so press enter now what we're looking for if we scroll down is we're looking for the line which says define and then a space and then DB host so this is the database host that WordPress attempts to connect to and currently it's set to local host which means it will use the database on the same EC2 instance as the application we're going to delete this local host so delete until we have two single quotes and then make sure that you still have the private IP version for address of this separate database instance in your clipboard if you don't just go ahead and copy it again from the EC2 console and then paste that in place of local host so now you should see DB underscore host and this now represents this private IP address and now the private IP address that you should use here will be different you need to use your private IP address of your a4l - DB - WordPress EC2 instance so now that you've updated this configuration file press control o and enter to save and then control x to exit out of editing this file now this now means that the WordPress instance is going to be communicating with the separate MariaDB database instance let's verify that let's go back to the tab that we have to our WordPress application and let's just go ahead and do a refresh if everything's working as expected we should see that the blog reloads successfully now this means that this blog is now pointing at this separate MariaDB database instance to be doubly sure of this though let's go back to the WordPress instance and let's shut down the MariaDB database server and we do that using this command so shudu space service space MariaDB space and then stop so type or copy and paste that command in and press enter and that's going to stop the MariaDB database service which is running on a4l WordPress so now the only MariaDB database that we have running is on the a4l - DB - WordPress EC2 instance now we can go back to the WordPress tab and hit refresh and assuming it loads in as it does in my case this now confirms that WordPress is communicating with this dedicated MariaDB EC2 instance now the reason why I wanted to step you through all these tasks in this demo lesson is the time a firm believer that in order to understand best practice architecture you need to understand bad architecture and as I mentioned in the theory lesson there is almost no justification for running your own self-managed database server on an EC2 instance in almost all situations it's preferable to use the RDS service but I need you to understand exactly how the architecture works when you're self managing a database and how to migrate from a monolithic all-in-one architecture through to having a separate self managed database in the demo lesson that's coming up next in the course you're going to migrate from this through to an RDS instance so that's step two but at this point you've done everything that I wanted you to do in this demo lesson you've implemented the architecture that's on screen now on the right all we need to do is to tidy up all of the infrastructure that we've used within this lesson so to do that it's nice and easy just go back to the cloud formation console make sure that you have the monolith to EC2 DB stack selected click on the delete button and then confirm that deletion and that stack deleting will clean up all of the infrastructure that we've used throughout this demo lesson and it will return the account into the same state as it was at the start of the lesson at this point you've completed all of the tasks that I want you to do so I hope you've enjoyed this demo lesson go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      Now one final thing before we finish with this demo lesson, and I want to talk about private hosted zones.

      So move back to the Route 53 console.

      I'm going to go to Hosted Zones, and I'm going to create a private hosted zone.

      So click "Create Hosted Zone" because it's a private hosted zone, it doesn't even need to be one that I actually own.

      So I'm going to call my hosted zone "IlikeDogsReally.com".

      It's going to be a private hosted zone.

      And for now, I'm going to associate it with the default VPC in US-East-1.

      So I'm going to pick the region, US-East, and then select Northern Virginia, and then click in the VPC ID box, and we should see two VPCs listed.

      One is the Animals for Life VPC, it's tagged A4L-VPC1, but I'm not going to pick this one, I'm going to pick the one without any text after it, which is the default VPC.

      So once that's set, I'm going to create the hosted zone.

      Then inside the hosted zone, I'm going to create a record.

      The record's going to use the simple routing policy.

      Click on "Next".

      I'm going to define a simple record.

      I'm going to call it "www".

      The record type is going to be "a routes traffic to an IP version 4 address and some resources".

      I'm going to click in this endpoint box and select IP address or another value, depending on record type.

      And then into this box, I'm just going to put a test IP address of 1.1.1.1.

      And then down at the bottom, I'm going to click "1M" to change this TTL to 60 seconds.

      And I'm going to click "Define simple record".

      And then finally, "Create records".

      So now we have a record called "www.ilikedogsrealy.com".

      So copy that into the clipboard.

      Move back to the EC2 console.

      Click on "Dashboard".

      Click on "Instances running".

      Right click, "Connect".

      We're going to use EC2 "Instance connect".

      And then just click on "Connect".

      Now once connected, I'm going to try pinging the record which I just created.

      So "Ping Space" and then paste in "www.ilikedogsrealy.com" and press "Enter".

      What you should see is "Name or service not found".

      The reason for this is the private hosted zone which we created is currently associated with the default VPC.

      And this instance is not in the default VPC.

      To enable this instance to resolve records inside this private hosted zone, we need to associate it with the "Animals for Life" VPC.

      So go back to the Route 53 console.

      Expand "Hoster Zone Details" and then "Edit hosted zone".

      Scroll down and we're going to add another VPC.

      In the region drop down, "US-East-1" and then in the "Choose VPC" box select "A4L-VPC-1".

      Scroll down and save changes.

      Now this might take a few seconds to take effect, but if we go back to the EC2 instance and try to run this ping again, and we still get "Name or service not found".

      So what I want you to do is go ahead and pause this video, wait for 4 or 5 minutes and then resume and try this command again.

      Now in my case it took about 5 minutes, but after a while I can now ping www.ilikedogsreally.com because I've now associated this private hosted zone with the VPC that this instance is running from.

      Now that's everything that I wanted to cover in this demo lesson, so all that remains is for us to clean up all of the infrastructure which we've created in this demo lesson.

      So if we go back to the Route 53 console and select "Health Checks", first we're going to delete the health check.

      So select "A4L Health" and click on "Delete Health Check" and confirm.

      Click on "Hostered Zones".

      Go inside the private hosted zone that you created.

      Select the www.ilikedogsreally.com record and then click on "Delete Record".

      Confirm that deletion.

      Go back to "Hostered Zones".

      Select the entire private hosted zone and click on "Delete".

      Type "Delete" and then click to confirm.

      And that will delete the entire private hosted zone.

      Then go inside the public hosted zone that you have.

      Select the two www records that you created earlier in this lesson.

      Click on "Delete Records".

      Click "Delete" to confirm.

      Then go to the S3 console.

      Click on the bucket that you created earlier in this lesson.

      Click "Empty".

      Copy and paste or type "Permanently Delete" and click on "Empty".

      Once that bucket is emptied click on "Exit".

      With it still selected click on "Delete".

      Copy and paste or type the full name into the box and click on "Delete Bucket".

      Then go to the EC2 console.

      Click the hamburger menu.

      Scroll down.

      Click "Elastic IPs".

      Select the elastic IP that you associated with the EC2 instance.

      Click on the actions drop down.

      Disassociate and then click to disassociate.

      With it still selected click on "Actions".

      Release elastic IP addresses and click on "Release".

      At that point all of the manually created infrastructure has been removed.

      Go back to the cloud formation console.

      Go to "Stacks".

      Select the stack that you created at the start of this lesson using the one click deployment.

      It should be called DNS and failover demo.

      Select it.

      Click on "Delete".

      Then click on "Delete Stack" to confirm that deletion.

      Once that's deleted the account will be back in the same state as it was at the start of the lesson.

      At this point that's everything I wanted to cover in this demo.

      I hope it's been enjoyable and it's given you some good practical experience of how to use failover routing and private hosted zones.

      That will be useful both for the exam and real world usage.

      At this point that's everything so go ahead and complete this video.

      When you're ready I'll look forward to you joining me in the next.

    1. Welcome to this demo lesson where you're going to get experience configuring fail-over routing as well as private hosted zones.

      Now with this demo lesson you have the choice of either following along in your own environment or watching me perform the steps.

      If you do wish to follow along in your own environment you will need a domain name that's registered within Route 53.

      Remember that was an optional step at the start of this course so if you did register a domain of your own then you can do this demo lesson.

      In my case I registered animals for life 1337.org.

      If you registered a domain it will be different and so wherever you see me use animals for life.org you need to replace it with your registered domain.

      If you didn't register one then you'll have to watch me perform all of these steps because you can't do this lesson without your own registered domain.

      In order to get started you need to make sure that you're logged in as the I am admin user of the general AWS account which is the management account of the organization and you'll need to have the Northern Virginia region selected.

      Now we're going to need to create some infrastructure in order to perform this demo lesson so attached to this lesson is a one-click deployment link and you should go ahead and click that link now.

      That's going to take you to a quick create stack screen.

      Everything should be pre-populated the stack name is DNS and failover demo all you'll need to do is scroll down to the bottom check this capabilities box and then click on create stack.

      That's going to take a few minutes and it's going to create infrastructure that we're going to need to continue with the demo lesson so go ahead and pause the video wait for your stack to move into a create complete state and then we're good to continue.

      Okay so the stacks now in a create complete state and it's created a number of resources the most important one being a public EC2 instance so we just need to test this first so if you just click in the search box and type EC2 and then right click to open that in a new tab.

      Once you're there click on instances running and you should see a4l-web just select that under public IP version 4 just click on this symbol to copy the IP address into your clipboard make sure you don't click open address because that's going to try and use HTTPS which we don't want so copy this IP address into your clipboard and then open that in a new tab and you should see the animals for life super minimal homepage and if you see that it means everything's working as intended so go ahead and close down that tab.

      Now we also need to give this instance an elastic IP address so that it has a static public IP version 4 address now to give it an elastic IP on the menu on the left scroll down to the bottom under network and security select elastic IPs and then we need to allocate an elastic IP make sure us - east - 1 is in this box scroll down and click on allocate once the elastic IP address is allocated to this account select it click on actions and then associate elastic IP once we're at this screen make sure instance is selected click in this search box and then select a4l-web once selected click in the private IP address box and select the private IP address of this instance and then check the box to say allow this elastic IP address to be re-associated once all that's complete click on associate and that now means that our EC2 instance has been allocated with a static IP version 4 address now we're configuring failover DNS and so the EC2 instance is going to be our primary record so we're going to assume that this is the animals for life main website and we want to configure an S3 bucket which is running as the backup in case this EC2 instance fails so the next thing we need to do is to create the S3 bucket so click in the search box type S3 and open that in a new tab and then go to the S3 console and at this point we're going to create an S3 bucket and configure it as a static website now the naming of the S3 bucket is important earlier in the course you should have registered a domain name in my case I registered animals for life 1337.org so I'm going to create a bucket with the name www.animalsforlife1337.org you need to create one which is called www.and then the domain name that you registered so I'm going to click and create bucket the bucket name is www.animalsforlife1337.org and it's going to be in the US east northern Virginia region which is US-East-1 then we're going to scroll down and we're going to need to uncheck block all public access because this bucket is going to be used to host a static website I'll need to acknowledge that I'm okay with that so I'll do that and then scroll all the way down to the bottom and then I'm going to click on create bucket then I'm going to go inside the bucket click on upload and then add files now attach to this lesson is an assets file I want you to go ahead and download that file then extract it and wave extracted it it should create a folder called R53 underscore zones underscore and underscore failover go inside that folder and there'll be two more folders one which is 01 underscore A4L website and another which is 02 underscore A4L failover we're interested in the A4L failover so go into that folder select both these files so index.html and minimal.jpeg click on open and then upload those files so we'll scroll down and click on upload once that's completed click on close then we go into enable static website hosting so click on properties and to enable this it's all the way down towards the bottom click on edit next to static website hosting and enable it make sure that host a static website is selected and then for the index document and the error document we're going to type index.html and once both of those are entered scroll down to the bottom and save changes now we've one final thing to do on this bucket we need to add a bucket policy so that this bucket is public so we need to click on permissions scroll down and then under bucket policy click on edit and this bucket currently does not have a bucket policy now also inside the assets folder that you extracted earlier in this lesson there's a file called bucket underscore policy.json this is the file so you'll need to copy the contents of that file into your clipboard and then inside this policy box paste that in and then click on the icon next to the bucket ARN to copy that into your clipboard and then we need to replace this placeholder with what you've just copied so I want you to select from just to the right of the first speech mark all the way through to before the forward slash so you should have ARN colon AWS colon S3 colon colon colon colon and then example bucket and then go ahead and paste the text from your clipboard which will overwrite that with the actual bucket ARN so it should look like this once you've got that scroll down and save the changes so now we have the failover website configured the static website running from the S3 bucket so now we need to go ahead and move to the route 53 console where we going to create a health check and configure the failover record so click in the search box type route 53 right click and open that in a new tab then click on health checks we're going to create a health check for the health check name type a4l health and it's going to be an end point health check scroll down we're going to specify the endpoint by IP address the protocols going to be HTTP and we need the IP address of the EC2 instance so if we go back to the EC2 console the EC2 instance is now using the elastic IP so if we scroll down and click on elastic IPs and copy the elastic IP into our clipboard then go back to the route 53 console and paste that in and the health check is going to be configured to health check the index.html document so in path we need to click and type index.html then we're going to expand advanced configuration and by default a health check is checking every 30 seconds so this is a standard health check we need to change this to fast because we want our health check to react as fast as possible if our primary website fails so select fast scroll down to the bottom click on next we don't want to create an alarm because we don't want to take any action if this health check fails we're just going to use it as part of our fail-over routing so go ahead and make sure no is selected and then click create health check now the health check is going to start off with an unknown status because it hasn't gathered enough information about the health of the primary website it's going to take a few minutes to move from this status to either healthy or unhealthy what we can do though is if we check this we can click on the health checkers tab and start to see the results of the globally distributed set of health check endpoints so we can see that we're already getting success HTTP status code 200 and this is telling us our primary website is already passing these individual checks and after a couple of minutes if we hit refresh we should see that the status changes from unknown to healthy so next we need to create the failover record so click on hosted zones locate the hosted zone for the domain that you registered at the start of the course and click it then click on create record now you can switch between two different modes either the quick create record mode or the wizard mode we're going to keep this demo simple so click on switch to wizard we're going to choose a failover record so select failover and click next we're going to call the record www we're going to set a TTL of one minute so click 1m and that will change the TTL seconds to 60 scroll down and we're going to define some failover records so click define failover record first we need to create the primary record so click in this first drop down and we're going to pick IP address or another value depending on record type so click that and then we need the elastic IP address so go back to the EC2 console and copy the elastic IP into your clip board and paste that into this box and then for failover record type this is the primary record so click on primary we need to associate it with a health check so click in that drop down and choose a for L health now once we do that it means that this primary record will only be returned if this health check is healthy otherwise the secondary record will be returned which we're going to define in a second under record ID just go ahead and type EC2 this needs to be unique within this set of records with the same name so going to call one EC2 and the other S3 so this one's EC2 so define that failover record and then we're going to define a new failover record so click that box again this time in this drop down we need to scroll down and we're going to select alias to an S3 website endpoint so select that choose the region and it needs to be US - East - 1 once selected you should be able to click in this box and see the S3 bucket that you just created so click on this to select that S3 bucket and we're going to set this as the secondary record so click on secondary we won't be associating this with a health check and we won't be evaluating the target health this record will only ever be used if the primary fails its health check and so we want this record to take effect whenever the health check associated with the primary fails and we're going to test that by shutting down the EC2 instance so this record should then take over so finally we need to enter S3 in the record ID and click on define failover record once we've done both of those we can go ahead and click on create records so now that we have both of those records in place the primary pointing at EC2 and the secondary at S3 if we copy down this full DNS name into our clipboard and open that in a new tab that should direct towards the animals for life.org super minimal homepage remember this is the website running on EC2 now what we need to do is to simulate a failure so go back to the EC2 console scroll to the top click on EC2 dashboard then instances running right click on this instance select stop instance and confirm that by clicking stop so now we've stopped this instance it should begin failing the health check so let's go back to the route 53 console click on health checks select this a4 health health check click on the health checkers tab and then click on refresh and over the coming seconds we should start to see some failure responses in this status column there we go we're getting connection timed out and over the next minute or so we should see that the status of the health check overall should move from healthy to unhealthy let's click on refresh it might take a minute or so for that to take effect so let's just give it a minute or so and now we can see that it's moved into an unhealthy state now this means that our failover record will detect this and then it's going to start returning the secondary record rather than the primary now DNS does have a cache remember we set the TTL value to 60 seconds so one minute but what we should find after that cache expires if we go back to this tab which we have open to the www.animalsforlife.org website and if we now hit refresh we should see that it changes to the animals for life.org super minimal failover page and this is the website that's running on s3 so the failover record has used a health check detected the failure of the EC2 instance and redirected us towards the backup s3 site so now we can go ahead and reverse that process if we go back to the EC2 console we can right click on this instance and start the instance that will take a few minutes to move from the stopped state through the pending state and then finally to running and once it's in a running state if we go back to the route 53 console and select this health check and then refresh on the health checkers initially we'll see a number of different messages if we keep hitting refresh over the next few minutes we should see this change to an okay message there we can see the first HTTP status code 200 if we keep refreshing we'll see more of those again more 200 statuses which means okay now that all of these are coming back okay let's click refresh on the health check itself it's still showing us unhealthy let's give it a few more seconds now it's reporting as healthy again if we go back to the tab that we have open to the website and click on refresh now it should change back to the original EC2 based website and it does so that means our failover record has worked in both directions it's failed over to s3 and failed back to EC2 okay so this is the end of part one of this lesson it was getting a little bit on the long side and so I wanted to add a break it's an opportunity just to take a rest or grab a coffee part 2 will be continuing immediately from the end of part one so go ahead complete the video and when you're ready join me in part 2.

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      So let's go back to the instance now.

      Just press enter a few times to make sure it hasn't timed out.

      That's good.

      Now there's a small bug fix that we need to do before we move on.

      The CloudWatch agent expects a piece of system software to be installed called CollectD.

      And on Amazon Linux that is not installed.

      So we need to do two things.

      The first is to create a directory that the agent expects to exist.

      So run that command.

      And the second is to create a database file that the agent also expects to exist.

      And we can do that by running this command.

      Now at this point we're ready to move on to the final step.

      So we've installed the agent and we've run the configuration wizard to generate the agent configuration.

      And we're now safely stored inside the parameter store.

      The final step is to start up the CloudWatch agent and provide it with the configuration that's stored inside the parameter store.

      And by doing that the agent can access the configuration.

      It can download it.

      It can configure itself as per that configuration.

      And then because we've got an attached instance role that has the permissions required, it can also inject all of the logging data for the web server and the system into CloudWatch logs.

      So the final step is to run this command.

      So this essentially runs the Amazon hyphen CloudWatch hyphen agent hyphen CTL.

      And it specifies a command line option to fetch the configuration.

      And it uses hyphen C and specifies SSM colon and then the parameter store parameter name.

      Essentially what this command does is to start up the agent, pull the config from the parameter store, make sure the agent is running and then it will start capturing that logging data and start injecting it into CloudWatch logs.

      So at this point if it's functioning correctly, what you should be able to do is go back to the AWS console, go to services, type CloudWatch and then select CloudWatch to move to the CloudWatch console.

      Then if we go to log groups, now you might see a lot of log groups here.

      That's fine.

      Every time you apply the animals for life VPC templates, it's actually using a Lambda function which we haven't covered yet to apply an IP version six workaround, which I'll explain later in the course when we cover Lambda.

      What you should find though is if you just scroll down all the way to the bottom, you should see either one, two or three of the log groups that we've created.

      In this example on screen now, you can see that I have /var/log/httbd/error_log.

      Now these logs will start to appear when they start getting new entries and those entries are sent into CloudWatch.

      So right now you can see that I only have the error log.

      Now if you don't see access underscore log, what you can do is go back to the EC2 consoles, select the WordPress instance that you've created using the one click deployment and then copy the public IP version four address into your clipboard.

      Don't use this link, just copy the IP address and then open that in a new tab.

      Now by doing that, it will generate some activity within the Apache web server and that will put some log items into the access log and that will mean that that logging information will then be injected into CloudWatch logs using the CloudWatch agent.

      So if we move back to CloudWatch logs and then refresh, scroll down to the bottom.

      Now we can see the access underscore log file.

      Open the log stream for the EC2 instance.

      This log file details any accesses to the web server on the EC2 instance.

      You won't have a lot of entries in this.

      Likewise, if you go back to log groups and look for the error log, that will detail any errors, any accesses which weren't successfully served.

      So if you try to access a web page which doesn't exist, if there's a server error or any module errors, these will show inside this log group.

      Now also, because we're using the CloudWatch agent, we also have access to some metrics inside the EC2 instance that we otherwise would not have had.

      If we click on metrics and just drag this up slightly so we can see it, you'll see the AWS namespaces.

      So these are namespaces with metrics inside that you would have had access to before, but there'll also be the CWAgent namespace and inside here, just maybe select the image ID, instance ID, instance type name.

      Inside there, you'll see all of the metrics that you now have access to because you have the CloudWatch agent installed on this EC2 instance.

      So these are detailed operating system level metrics such as disk, IO, read and write, and you would have not had access to these before installing the agent.

      If we select another one, image ID, instance ID, instance type CPU, we'll be able to see the CPU cores that are on this instance together with the IO weight and the user values.

      Again, these are things that you would not have had access to at this level of detail without the CloudWatch agent being installed.

      Now I do encourage you to explore all of the different metrics that you now have access to as well as to how the log groups and log streams look with this agent installed.

      But this is the end of what I had planned for this demo lesson.

      So as always, we want to clear up all of the infrastructure that we've created within this demo lesson.

      So to do that, I want you to move back to the EC2 console, right click on this instance, go down to security, select modify IAM role, and then remove the CloudWatch role from this instance.

      You'll need to confirm that by following the instructions.

      So to detach that role, then click on services and move back to IAM, click on roles.

      And I want you to remove the CloudWatch role that you created earlier in this demo.

      So select it and then click delete role.

      You'll need to confirm that deletion.

      What we're not going to do is delete the parameter value that we've created.

      So if we go to services and then move back to systems manager, go to parameter store, because this is a standard parameter.

      This won't incur any charges.

      And we're going to be using this later on in future lessons of this course and other courses.

      So this is a standard configuration for the CloudWatch agent, which we'll be using elsewhere in the course.

      So we're going to leave this in place.

      The last piece of cleanup that you'll need to do is to go back to the CloudFormation console.

      You should have the single CW agent stack in place that you created at the start of the demo using the one click deployment.

      Go ahead and select the stack, click on delete, and then confirm that deletion.

      And once that's completed, all of the infrastructure you've used in this demo will be removed and the account will be back in the same state as it was at the start of this demo.

      Now that's everything that I wanted you to do in this demo.

      I just wanted to give you a brief overview of how to manually install the CloudWatch agent within an EC2 instance.

      Now there are other ways to perform this installation.

      You can use systems manager or bake it into AMIs or you can bootstrap it in using the process that you've seen earlier in the course.

      We're going to be using the CloudWatch agent during future demos of the course to get access to this rich metric and logging information.

      So most of the demos which follow in the course will include the CloudWatch agent configuration.

      At this point though, that is everything I wanted you to do in this demo.

      Go ahead, complete this video, and when you're ready, I'll look forward to you joining me in the next. in the next.

    1. Welcome back and welcome to this demo where together we'll be installing the CloudWatch agent to capture and inject logging data for three different log files into CloudWatch logs as well as giving us access to some metrics inside the OS that we wouldn't have otherwise had visibility of.

      So it's going to be a really good demonstration to show you the power of CloudWatch and CloudWatch logs when combined with the CloudWatch agent.

      Now in order to do this demo you're going to need to deploy some infrastructure.

      To do so just make sure that you're logged in to the general AWS account, so the management account of the organization and as always make sure you've got the Northern Virginia region selected.

      Now attached to this lesson is a one-click deployment URL which will deploy the infrastructure that you'll be using during this demo.

      So go ahead and click on that link.

      This will take you to a quick create stack screen.

      The stack name should be pre-populated with CW agent.

      You just need to scroll all the way down to the bottom, acknowledge the capabilities and click on create stack.

      Also attached to this lesson is a lesson commands document which will contain all of the commands you'll be using during this demo lesson.

      So go ahead and open that in a new tab.

      Now you're going to need to let this cloud formation stack move into a create complete state before you continue the demo.

      So go ahead and pause the video, wait for the status to change to create complete and then you're good to continue.

      Okay so now this stack is in a create complete state then we good to continue the demo.

      Now during this demo lesson you're going to be installing the cloud watch agent on an EC2 instance and this EC2 instance has been provisioned by this one-click deployment.

      So the first thing that we need to do is to move across to the EC2 console and connect to this instance.

      Once you're at the EC2 console click on instances running.

      You should see one single EC2 instance called A4L WordPress.

      Just go ahead and select this, right-click on it, select connect.

      We're going to connect into this instance using EC2 instance connect so make sure that's selected.

      Make sure also that the username is set to EC2-user and then connect into the instance.

      Now if everything's working as it should be you should see the animals for life custom login banner when you log into the instance.

      In my case I do see that and that means everything's working as expected.

      So this demonstration is going to have a number of steps.

      First we need to download the cloud watch agent then we need to install the agent then we need to generate the configuration file that this install of the agent as well as any future installs of the agent could use and then we need to get the cloud watch agent to read this config and start capturing and injecting those logs into cloud watch logs.

      So step one is to download and install the agent and the command to do this is inside the lesson commands document which is attached to this lesson and that will install the agent but crucially it won't start it.

      What we need to do before we can start the agent is to generate the config file that we'll use to configure this and any future agents but because we also want to store that config file inside the parameter store and because we also want to give this instance permissions to interact with cloud watch logs before we continue we need to attach an IAM role to this instance an EC2 instance role so that's the next step.

      So we need to move back to the EC2 console click on services and then open the IAM console because we'll be creating an IAM role to attach to this instance you'll need to go to roles create role it'll be an AWS service role using EC2 so select EC2 and then click on next we'll need to attach two managed policies to this role so I've included the names of those managed policies in the lesson commands document the first is cloud watch agent server policy so make sure you type that in the filters policy and then check that box and the second is Amazon SSM full access so type that in the box select that policy and then scroll down and click on next and we'll call this role cloud watch role so enter cloud watch role and click on create role and once we've done that we can attach this role to our EC2 instance so we need to move back to the EC2 console go to instances right click on the instance go to security and then modify IAM role and then click on the drop down and select cloud watch role which is the role that you've just created then click update IAM role now that we've allocated that instance with the permissions that it needs to perform the next set of steps go ahead and connect to that instance again the tab may have timed out that you previously had open so if it doesn't respond you need to close it down and reopen it if it does respond that's fine keep the existing tab once we back in the terminal for the instance we need to start the cloud watch agent configuration wizard and the command to do that is also in the lesson commands document attached to this lesson so go ahead and paste that in and press enter and that will start off the configuration wizard for the cloud watch agent now for most of these values we can accept the defaults but we need to be careful because there are a number of them that we can't so press enter and accept the default for the operating system it should automatically detect Linux press enter it should automatically detect that it's running on an EC2 instance press enter to use the root user again that should be the default press enter for stats D press enter for the stats D port press enter for the interval press enter for the aggregation interval press enter to monitor metrics from collect D again that's the default press enter to monitor host metrics so CPU and memory press enter to monitor CPU metrics per core press enter for the additional dimensions press enter to aggregate EC2 dimensions press enter for the default resolution so 60 seconds for the default metric config that you want the default will be basic go ahead and enter 3 for advanced this captures additional operating system metrics that we might actually want so use 3 for this value press enter to indicate that we satisfied with the above config next we'll move to the log configuration part of this wizard so press enter for the default of no we don't have an existing cloud watch log agent config to import press enter which is the default for yes we do want to monitor log files you'll be asked for the log file path to monitor so the first one that we want to monitor and again these are in the lesson commands document so the first log path is forward slash var forward slash log forward slash secure press enter you'll be asked for the log group name the default is just the log name itself so secure but we're going to enter the full path I always prefer using the full path for the log group names for any system logs so going to enter var log secure again you'll be asked for the log stream name remembering the theory part of this lesson I talked about how a log stream will be named after the instance which is injecting those logs so the default choice is to do that to use the instance ID so press enter it's here where you can specify a log group retention in days we're just going to accept the default for the log group retention value the default will be yes we do want to specify additional log files so press enter the log file path for this one will be var log HTTP d access underscore log so enter that the log group name again will default to the name of the actual log we want the full path so enter the full path again press enter the log stream name the default for this is again the instance ID which is fine just press enter go ahead and accept the default for the log group retention in days press enter again we've got one more log file that we want to enter this time the log file path is var log HTTP d error underscore log again the log group name will default the name of the actual log we want to use the full path so enter the same thing again the default choice for log stream name will be again the instance ID that's fine press enter go ahead and accept the default for the log group retention in days and now we finished adding log files we won't want to log any additional files so press 2 and that will complete this logging section of this wizard it's asking us to confirm that we're happy with this configuration file and it's telling us that the configuration file is stored at forward slash opt forward slash aws forward slash amazon hyphen cloud watch hyphen agent forward slash bin and then config dot json in that folder now that's where it stores it on the local file system but we can also elect to store this json configuration inside the parameter store and I thought that since we've previously talked about the theory of the parameter store and done a little bit of interaction it would be useful for you to see exactly how it can be used in a more production like setting so the default is to store the configuration in the parameter store so we're going to allow that press enter it'll ask us for the parameter name to use and the default is Amazon cloud watch hyphen linux and that's fine so press enter it'll ask us for the region to use because parameter store is like many other services a regional service and the default region is the one where the instance is in so it automatically detects that we're in us east one which is northern Virginia so go ahead and accept that default choice it'll ask us for the credentials that it can use to send that configuration into the parameter store now these credentials will be obtained from the role that we've attached to this instance in the previous step so you can accept the default choice it'll use those credentials to store that configuration inside the parameter store and if we move back to the ec2 console and we switch back to the parameter store so just type SSM to move to systems manager which is the parent product of parameter store if we go down to the parameter store item on the menu on the left we'll be able to see this single parameter Amazon cloud watch hyphen linux and if we open that up and just scroll down we can see that the value is a JSON document with the full configuration of the cloud watch agent so we can now use this parameter to configure the agent on this ec2 instance as well as any other ec2 instances we want to deploy so if you create the cloud watch configuration once and then store it into parameter store then when you create ec2 instances at scale as you'll see how to do later in the course when we talk about auto scaling groups then you can use the parameter store to deploy this type of configuration at scale in a secure way okay so this is the end of part one of this lesson it was getting a little bit on the long side and so I wanted to add a break it's an opportunity just to take a rest or grab a coffee part 2 will be continuing immediately from the end of part one so go ahead complete the video and when you're ready join me in part 2.

    1. Welcome back and in this demo lesson, I'm just wanting to give you some practical experience with interacting with the parameter store inside AWS.

      So to do that, make sure you're logged into the IAM admin user of the management account of the organization and you'll need to have the Northern Virginia region selected.

      Now there's also a lesson commands document linked to this lesson, which contain all of the commands that you'll need for this lessons demonstration.

      So before we start interacting with the parameter store from the command line, we need to create some parameters.

      And the way that we do that is first move to systems manager.

      So the parameter store is actually a sub product of systems manager.

      So move over to the systems manager console.

      And once you're there, you'll need to select the parameter store from the menu on the left.

      So it should be about halfway down on the left and it's under application management.

      So go ahead and select parameter store.

      Now once you're in parameter store, the first thing that you'll need to do to remove this default welcome screen logically is to create a parameter.

      So go ahead and click on create parameter.

      Now when you create a parameter, you're able to pick between standard or advanced.

      Standard is the default and that meets most of the needs that most people have for the product.

      And you can create up to 10,000 parameters using the standard tier.

      With the advanced tier, you can create more than 10,000 parameters.

      The parameter value can be longer at eight kilobytes versus the four kilobytes of standard.

      And you do gain access to some additional features.

      But in most cases, most parameters are fine using the default, which is the standard tier.

      With the standard tier, there's no additional charge to use this up to the limit of 10,000 parameters.

      The only point at which parameter store costs any extra is if you use the faster throughput options or make use of this advanced tier.

      And we won't be doing that at any point throughout the course.

      We'll only be using standard.

      And so there won't be any extra parameter store related charges on your bill.

      Now I mentioned that a parameter is essentially a parameter name and a parameter value.

      And it's here where you set both of those.

      There's an optional description that you can use and you can set the type of the parameter.

      The options being string, string list, which is a comma separated list of individual strings and then secure string, which utilizes encryption.

      So we're gonna go ahead at this point and create some parameters that we're then going to interact with from the command line.

      So the first one we'll create is one that's called forward slash my-cat-app forward slash DB string.

      So this is the name of a parameter and it will also establish a hierarchy.

      So anytime we use forward slashers, we're establishing a hierarchy inside the parameter store.

      So imagine this being a directory structure.

      Imagine this being the root of the structure.

      Imagine my-cat-app being the top level folder and inside there, imagine that we've got a file called DB string.

      So we're going to store this, we're going to store this hierarchy and we need to set its value.

      So we'll keep this for now as a string and this is going to be the database connection string for my-cat-app.

      So we'll just enter the value that's in the lesson commands document.

      So DB dot all the cats dot com colon 3306.

      And 3306 of course is the my SQL standard port number.

      At this point, we could enter an optional description.

      So let's go ahead and do that.

      Connection string for cat application.

      So just type in a description here.

      It doesn't matter really what you type and then scroll down and hit create parameter.

      So that's created our first parameter, my-cat-app forward slash DB string.

      Now we're going to go ahead and do the same thing but for DB user.

      So click on create parameter and then just click in this name box and notice how it presents you with this hierarchy.

      So now we've got two levels of this hierarchical structure.

      We've got the my-cat-app at the top and then we've got the actual parameter that we created at the bottom here.

      So this has already established this structure.

      So let's go ahead and create a new parameter.

      This time it's going to be forward slash my-cat-app forward slash DB user.

      We'll not bother with the description for this one.

      We'll keep it at the default of standard and it will also be a string.

      And then for the value, it'll be boss cat.

      So enter all that and click on create parameter.

      Next let's create a parameter again.

      If we click in this name this time, we've got this hierarchy that's ever expanding.

      So we've got the top level at the top and then below it two additional parameters, DB string and DB user.

      And we're going to create a third one at this level.

      So this time it's going to be called forward slash my-cat-app forward slash DB password.

      This time though, instead of type string, it's going to be a secure string so that it encrypts this parameter.

      And it's going to use KMS to encrypt the parameter.

      And because it's using KMS, we'll need to select the key to use to perform the cryptographic operations.

      We can either select a key from the current account, so the account that we're in, or we can select another AWS account.

      And in either case, we'll need to pick the key ID to use and by default, it uses the product default key for SSM.

      So that's using alias forward slash AWS forward slash SSM.

      And you always have the option of clicking on this dropdown and changing it if you want to benefit from the extra functionality that you get by using a customer managed KMS key.

      This is an AWS managed one.

      So you won't be able to configure rotation and you won't be able to set these advanced key policies.

      But in most cases, you can use this default key.

      So at this point, we'll leave it as the default and we'll enter our super secret password, amazing secret password, 1337, and then click create parameter.

      We're not finished yet though, click on create parameter again.

      And I like to be inclusive, so not everything in my course is going to be about cats.

      We're going to create another parameter, my-dog-app forward slash DB string.

      We'll keep standard, we'll keep the type as string and then the value for connecting to the my-dog application.

      So the DB string is going to be DB if we really must have dogs.com colon 3306.

      So type that in and then click on create parameter.

      And then lastly, we're going to create one more parameter.

      This time the name is going to be forward slash rate my lizard.

      So rate hyphen my hyphen lizard forward slash DB string.

      The tier is going to be standard again.

      The type is going to be string.

      And for the value, it will be DB.

      This is pretty random.com colon 3306.

      So type that in and then click on create parameter.

      So now we've created a total of five parameters.

      We've created the DB string, the DB user, and the DB password for the cat application.

      And then the DB string for the dog application as well as the rate my lizard application.

      So a total of five parameters and one of them is using encryption.

      So that's the DB password for the my cat application.

      So now let's switch over to the command line and interact with these parameters.

      And to keep things simple, we're going to use the cloud shell.

      So this is a relatively new feature made available by AWS.

      And this means that we don't have to interact with AWS using our local machine.

      We can do it directly from the AWS console.

      So click on the cloud shell icon on the menu on the top.

      This will take a few moments to provision because this is creating a dedicated environment for you to interact with AWS using the command line interface.

      So you'll need to wait for this process to complete.

      So go ahead and pause the video and wait until this logs you into the cloud shell environment at which point you can resume the video and we're good to continue.

      It'll say preparing your terminal and then you'll see a familiar looking shell much like you would if you were connected to a Linux instance.

      And now you'll be able to interact with AWS using the command line interface, using the credentials that you're currently logged in with.

      Now to interact with parameter store using the command line, we start by using AWS and then a space, SSM, and then a space, and then the command that we're going to use is get-parameters.

      Now by default, what we need to provide the get-parameters command with is the path to a parameter.

      So in this case, if we wanted to retrieve the database connection string for the rate my lizard application, then we could provide it with this name.

      So forward slash rate-my-lizard/dvstring.

      And this directly maps back through to the parameter that we've just created inside the parameter store.

      So this parameter.

      So if you go ahead and type that and press enter, it's going to return a JSON object.

      Inside that JSON object is going to be a list of parameters and then for each parameter, so everything inside these inner curly braces, we're going to see the name of the parameter that we wanted to retrieve, the type of the parameter, the value of the parameter.

      In this case, db.this_is_pretty_random.com/3306, the version number of the parameter because we can have different version numbers, the last modified date, the data type, and then the unique arn of this specific parameter.

      And so this is an effective way that you can store and retrieve configuration information from AWS.

      Now we can also use the same structure of command to retrieve all of those other parameters that we stored within the parameter store.

      So if we wanted to get the db string for the my-dog-app, then we could use this command.

      And again, it would return the same data structure.

      So a JSON object containing a list of parameters and each of those parameters would contain all of this information.

      I'll clear the screen to keep this easy to see.

      We could do the same for the my-cat-app, retrieving its database connection string.

      And again, it would return the same JSON object with the parameters list.

      And then for each parameter, this familiar data structure.

      Now what you can also do, and I'm going to clear the screen before I run this, is instead of providing a specific path to a parameter.

      So if you remember, we had almost a hierarchy that we created with these different names.

      So we have the my-cat-app hierarchy and then inside there db-pastword, db-string, db-user.

      We have my-dog-app and inside there db-string and then rate-my-lizard and also db-string.

      So rather than having to retrieve each of these individual parameters by specifying the exact name, we can actually use get parameters by path.

      So let's demonstrate exactly how that works.

      So with this command, we're doing a get-parameters-by-path and we're specifying a path to a group of a number of parameters.

      So in this case, my-cat-app is actually going to be the first part of the path of db-pastword, db-string and db-user.

      So by creating a hierarchical structure inside the parameter store, we can retrieve multiple parameters at once.

      So this time we're returning a JSON structure.

      Inside this JSON structure, we have a list of parameters and then we're retrieving three different parameters, db-pastword, db-string and db-user.

      Now note how db-pastword is actually a type of secure string and by default, if we don't specify anything, we return the encrypted version of this parameter.

      So the ciphertext version of this parameter.

      This ensures that we can interact with parameters without actually decrypting them and this offers several security advantages.

      Now I've cleared the screen to make this next part easy to see because it's very important.

      Because we're using KMS to encrypt parameters, the permissions to access KMS keys to perform this decryption, this is separate than the permissions to access the parameter store.

      So if this user, I am admin in this case, has the necessary permissions to interact with KMS to use the keys to decrypt these parameters, then we can also ask the parameter store to perform that decryption whilst we retrieve the parameters.

      The important thing to understand is the permissions to interact with the parameter store are separate than the permissions to interact with KMS.

      So to perform a decryption whilst we're retrieving the parameters, we would use this command.

      So it's the same command as before, aws, SSM, get-parameters-by-path, and then we're specifying the my-cat-app part of the hierarchy.

      So remember, this represents these three parameters.

      Now, if we ran just this part on its own, which was the command we previously ran, this would retrieve the parameters without performing decryption.

      But by adding this last part, this is the part that performs the decryption on any parameter types which are encrypted.

      And if you recall, one of the parameters that we created was this DB password, which is encrypted.

      So if we run this command, this time it's going to retrieve the /my-cat-app/db password parameter, but it's going to decrypt it as part of that retrieval operation and return the plain text version of this parameter.

      And just to reiterate, that requires both the permissions to interact with the parameter store, as well as the permissions to interact with the KMS key that we chose when creating this parameter.

      Now, we're logged in as the IAM admin user, which has admin permissions, and so we do have permissions on both of those, on SSM and on KMS, so we can perform this decryption operation.

      Now, you're going to be using the parameter store extensively for the rest of the course and my other courses.

      It's a great way of providing configuration information to applications, both AWS and bespoke applications within the AWS platform.

      It's a much better way to inject configuration information when you're automatically building applications or you need applications to retrieve their own configuration information.

      It's much better to retrieve it from the parameter store than to pass it in using other methods.

      So we're going to use it in various different lessons as we move throughout the course.

      In this demo lesson, I just wanted to give you a brief experience of working with the product and the different types of parameters.

      But at this point, let's go ahead and clear up all of the things that we've created inside this demo lesson.

      So close down this tab, open to Cloud Shell.

      Back at the parameter store console, just go ahead and check the box at the top to select all of these existing parameters.

      If you do have any other parameters, apart from the ones that you've created within this demo lesson, then do make sure that you uncheck them.

      You should be using an account dedicated for this training, so you shouldn't have any others at this point.

      But if you do, make sure you uncheck them.

      You should only be deleting ones for 8-mile lizard, my dog app, and my cat app.

      So make sure that all of those are selected and then click on delete to delete those parameters, and you'll need to confirm that deletion process.

      And at this point, that's everything that I wanted you to do in this lesson.

      You've cleared up the account back to the point it was at the start of this demo lesson.

      So go ahead, complete this video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome to this demo lesson where you're going to get the experience of working with EC2 and EC2 instance roles.

      Now as you learned in the previous theory lesson, an instance role is essentially a specific type of IAM role designed so that it can be assumed by an EC2 instance.

      When an instance assumes a role which happens automatically when the two of them are linked, that instance and any applications running on that instance can gain access to the temporary security credentials that that role provides.

      And in this demo lesson you're going to get the experience of working through that process.

      Now to get started you're going to need some infrastructure.

      Make sure that you're logged in to the general AWS account, so that's the management account of the organization and as always you'll need to be within the Northern Virginia region.

      Assuming you are, there's a one click deployment link which is attached to this lesson so go ahead and click that link.

      That will take you to a quick create stack page.

      The stack name will be pre-populated with IAM role demo and all you need to do is to scroll down to the bottom, check this capabilities box and then click on create stack.

      This one click deployment will create the animals for live VPC and EC2 instance and an S3 bucket.

      Now in order to continue with this demo we're going to need this stack to be in a create complete state.

      So go ahead and pause the video and then when the stack moves into a create complete status then we're good to continue.

      Okay so this stacks now in a create complete state and we're good to continue.

      So to do so go ahead and click on the services drop down and then type EC2, locate it, right click and then open that in a new tab.

      Once you're at the EC2 console click on instances running and you should be able to see that we only have the one single EC2 instance.

      Now we're going to connect to this to perform all the tasks as part of this demo.

      So right click on this instance, select connect, we're going to use EC2 instance connect.

      Just verify that the username does say EC2-user and then click on connect.

      Now the AMI that we use to launch this instance is just the standard Amazon Linux 2 AMI.

      And so if we type AWS and press enter it comes with the standard install of the AWS CLI version 2.

      Now it's important to understand that right now this instance has no attached instance role and it's not been configured in any way.

      It's the native Amazon Linux 2 AMI that's been used to launch this instance.

      And so if we attempt to interact with AWS using the command line utilities, for example by running an AWS S3 LS, the CLI tools will tell us that there are no credentials configured on this instance and will be prompted to provide long term credentials using AWS Configure.

      Now this is the method that you've used to provide credentials to your own installed copy of the CLI tools running on your local machine.

      So you've used AWS Configure and set up two named configuration profiles.

      And the way that you provide these with authentication information is using access keys.

      Now this instance has no access keys configured on it and so it has no method of interacting with AWS.

      We could use AWS Configure and provide these credentials but that's not best practice for an EC2 instance.

      What we're going to do instead is use an instance role.

      So to do that you're going to need to move back to the AWS console.

      And once you're there click on services and in the search box type IAM.

      We're going to move to the IAM console so right click and open that in a new tab.

      As I mentioned earlier an instance role is just a specific type of IAM role.

      So we're going to go ahead and create an IAM role which our instance can assume.

      So click on roles and then we're going to go ahead and click on create role.

      Now the create role process presents us with a few common scenarios.

      We can create a role that's used by an AWS service, another AWS account, a web identity or a role designed for SAML 2.0 Federation.

      In our case we want a role which can be assumed by an AWS service specifically EC2.

      So we'll select the type of trusted entity to be an AWS service then we'll click on EC2 and then we'll click on next.

      Now for the permissions in this search box just go ahead and type S3 and we're looking for the Amazon S3 read only access.

      So there's a managed policy that we're going to associate with this role.

      So check the box next to Amazon S3 read only access and then we'll click on next.

      And then under role name we're going to call this role A4L instance role.

      So it's easy to distinguish from any other roles we have in the account.

      So go ahead and enter that and click on create role.

      Now as I mentioned in the theory lesson about instance roles when we do this from the user interface.

      It's actually created a role and an instance profile of the same name and it's the instance profile that we're going to be attaching to the EC2 instance.

      Now from a UI perspective both of these are the same thing.

      You're not exposed to the role and the instance profile as separate entities but they do exist.

      So now we're going to move back to the EC2 console and remember currently this instance has no attached instance role and we're unable to interact with AWS using this EC2 instance.

      To attach an instance role using the console UI right click, go down to security and then modify IAM role.

      Select that and we'll need to choose a new IAM role.

      You have the option of creating one directly from this screen but we've already created the one that we want to apply.

      So click in the drop down and select the role that you've just created.

      In this case A4L instance role.

      So select that and then click on save.

      Now if we select the instance and then click on security you'll be able to confirm that it does have an IAM role attached to this instance.

      So this is the instance role that this EC2 instance can now utilize.

      So now we're going to interact with this instance again from the operating system.

      Now if it's been a few minutes since you've last used instance connect you might find when you go back it appears to have frozen up.

      If that's the case that's no problem just close down these tabs that you've got connected to that instance.

      Right click on the instance again, select connect, make sure the username is EC2-user and then click on connect.

      And this will reconnect you to that instance.

      Now if you recall last time we were connected we attempted to run AWS S3LS and the command line tools informed us that we had no credentials configured.

      Let's attempt that process again.

      AWS space S3 space LS and press enter.

      And now because we have the instance role associated with this EC2 instance the command line tools can use the temporary credentials that that role generates.

      Now the way that this works and I'm going to demonstrate this using the curl utility these credentials are actually provided to the command line tools via the metadata.

      So this is actually the metadata path that the command line tools use in order to get the security credentials.

      So the temporary credentials that a role provides when it's assumed.

      So if I use this command and press enter you'll see that it's actually using this role name.

      So you'll see a list of any roles which are associated with this instance.

      If we use the curl command again but this time on the end of security credentials we specify the name of the role that's attached to this instance and press enter.

      Now we can see the credentials that command line tools are using.

      So we have the access key ID the secret access key and the token and all of these have been generated by this EC2 instance assuming this role because these are temporary credentials.

      They also have an expiry date.

      So in my case here we can see that these credentials expire on the 7th of May 2022 at 552 47 UTC.

      And that really is all I wanted to show you in this demo lesson about instance roles.

      Essentially you just need to create an instance role and then attach it to an instance.

      And once you do that instance is capable of assuming that role gaining access to temporary credentials and then any applications installed on that instance, including the command line utilities are capable of interacting with AWS using those credentials.

      Now the process of renewing these credentials is automatic.

      So as long as the application that's running on the instance periodically checks the metadata service, it will always have access to up to date and valid credentials.

      The EC2 service once this expiry date closes in and once the expiry date is in the past, these credentials will be renewed and a new valid set of credentials will automatically be presented via the metadata service to any applications running on this EC2 instance.

      Now just one more thing that I do want to show you before we finish up with this demo lesson.

      And I have made sure that I've attached this link to the lesson.

      This link shows the configuration settings and precedence that the command line utilities use in order to interact with AWS.

      So whenever you use the command line interface, each of these is checked in order.

      First, it looks at command line options.

      Then it looks at environment variables to check whether any credentials are stored within environment variables.

      Then it checks the command line interface credentials file.

      So this is stored within the dot AWS folder within your home folder and then a file called credentials.

      Next, it checks the CLI configuration file.

      Next, it checks container credentials.

      And then finally, it checks instance profile credentials.

      And these are what we've just demonstrated.

      Now, this does mean that if you manually configure any long term credentials for the CLI tools as part of using AWS Configure, then they will be used as a priority over an instance profile.

      But you can use an instance profile and attach this to many different instances as a best practice way of providing them with access into AWS products and services.

      So that's really critical to understand.

      But at this point, that is everything that I wanted to cover in this demo lesson.

      And all that remains is for us to tidy up the infrastructure that we've used as part of this demo.

      So to tidy up this infrastructure, I want you to go back to the IAM console.

      I want you to click on roles and I want you to delete the A4L instance role that you've just created.

      So select it and then click on delete role.

      Once you've deleted that role, go back to the EC2 console, click on instances, right click on public EC2, go to security, modify IAM role.

      Now, even though you've deleted the IAM role, note how it's still listed.

      That's because this is an instance profile.

      This is showing the instance profile that gets created with the role, not the role itself.

      So what we're going to do, and I just wanted to do this to demonstrate how this works, we're just going to select no IAM role and then click on save.

      We'll need to confirm that.

      So to do that, we need to type detach into this box and then confirm it by clicking detach.

      That removes the instance role entirely from the instance.

      And then we can finish up the tidy process by moving back to the cloud formation console.

      Selecting the IAM role demo stack and then clicking on delete and confirming that deletion.

      And that will put the account back in the same state as it was at the start of this demo lesson.

      So this has been a very brief demo.

      I just wanted to give you a little bit of experience of working with instance roles.

      So that's EC2 instances combined with IAM roles in order to give an instance and any applications running on that instance, the ability to interact with AWS products and services.

      And this is something that you're going to be using fairly often throughout the course, specifically when you're configuring any AWS services to interact with any other services on your behalf.

      That's a common use case for using IAM roles and we'll be using instance roles extensively to allow our EC2 instances to interact with other AWS products and services.

      But at this point, that is everything that I wanted to cover in this demo lesson.

      So go ahead, complete the video and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back and I've mentioned a few times now within the course that I am roles are the best practice way that AWS services can be granted permissions to other AWS services on your behalf.

      Allowing a service to assume a role grants the service the permissions that that role has.

      EC2 instance roles are roles that an instance can assume and anything running in that instance has the permissions that that role grants and there is some detail involved which matters so let's take a look at how this feature of EC2 works architecturally.

      Instance role architecture isn't really all that complicated it starts off with an I am role and that role has a permissions policy attached to it so whoever assumes the role gets temporary credentials generated and those temporary credentials give the permissions that that permissions policy would grant.

      Now an EC2 instance role allows the EC2 service to assume that role which means there's an EC2 instance itself can assume it and gain access to those credentials but we need some way of delivering those credentials into the EC2 instance so that applications running inside that instance can use the permissions that the role provides so there's an intermediate piece of architecture the instance profile and this is a wrapper around an I am role and the instance profile is the thing that allows the permissions to get inside the instance when you create an instance role in the console an instance profile is created with the same name but if you use the command line or cloud formation you need to create these two things separately when using the UI and you think you're attaching an instance role direct to an instance you're not you're attaching an instance profile of the same name it's the instance profile that's attached to an EC2 instance.

      We know by now that when I am roles are assumed you're provided with temporary security credentials which expire but these credentials grant permissions based on the roles permissions policy will inside an EC2 instance these credentials are delivered via the instance metadata.

      An application running inside the instance can access these credentials and use them to access AWS resources such as S3.

      One of the great things about this architecture is that the credentials available inside the metadata they're always valid EC2 and the secure token service liaise with each other to ensure that the credentials are always renewed before they expire as long as your application inside the EC2 instance keeps checking the metadata it will never be in a position where it has expired credentials.

      So to summarize when you use EC2 instance roles the credentials are delivered via the instance metadata specifically inside the metadata there's an IAM tree in there there's a security credentials part and then in there is the role name and if you access this you'll get access to these temporary security credentials and they're always rotated they're always valid as long as that instance role remains attached to the instance anything running in the instance will always have access to these valid credentials applications running in the instance of course need to be careful about caching these credentials and just check the metadata before the credentials expire or do it periodically.

      You should always use roles where possible I'm going to keep stressing that throughout the course it's important for the exam roles are always preferable than storing long-term credentials so access keys into an EC2 instance it's never a good idea to store long-term credentials such as access keys anywhere which aren't securely stored so for example on your local machine.

      In fact the AWS tooling such as the CLI tools will use instance role credentials automatically so as long as the instance role is attached to the EC2 instance any command line tools running inside that instance can automatically make use of those credentials.

      So at this point that's everything I wanted to cover thanks for watching go ahead and complete this video and when you're ready join me in the next lesson.

    1. Welcome back and in this brief demonstration you'll have the opportunity to create an EC2 instance with WordPress bootstrapped in ready and waiting to be configured.

      But this time you'll be using an enhanced CloudFormation template which uses CFN init and creation policies rather than the simple user data that you used in the previous demonstration.

      To get started just make sure you are logged in to the general AWS account as the I am admin user and as always make sure you've got the northern Virginia region selected.

      Now attached to this lesson are two one click deployment links.

      Go ahead and use the first one which is the VPC link.

      Everything should be pre-populated.

      All you'll need to do is scroll down to the bottom, check the acknowledgement box and click on create stack.

      Once it's moved into a create complete status you can resume and we'll carry on with the demo.

      I'll assume that that's now in a create complete status and now we're going to apply another CloudFormation template.

      This is the template that we'll be using.

      It's just an enhancement of the one that you used in the previous lesson.

      This time instead of using a set of procedural instructions, so a script that are passed into the user data, this uses the CFN init system and creation policies.

      So let's have a look at exactly what that means.

      If I scroll down and locate the EC2 instance logical resource, then here we've got this creation policy.

      This means that CloudFormation is going to create a hold point.

      It's not going to allow this resource to move into a create complete status until it receives a signal.

      And it's going to wait 15 minutes for this signal.

      So a timeout of 15 minutes.

      Now scrolling down and looking at the user data, the only things we do in a procedural way, we use the CFN init command to begin the desired state configuration.

      That will either succeed or not.

      And based on that we use the CFN signal command to pass that success or failure state back to the CloudFormation stack.

      And that's what uses this creation policy.

      So the creation policy will wait for a signal and it's this command which provides that signal, either a success signal or a failure signal.

      Now what we're interested in specifically for this demo lesson is this CFN init command.

      So this is the thing that pulls the desired state configuration from the metadata of this logical resource.

      I'll talk all about that in a second.

      But it pulls that down by being given the stack ID and it uses this substitution command.

      So instead of this being passed into the instance, what's actually passed instead of this variable name, so the stack ID variable name, is the actual stack ID.

      And then likewise, instead of this variable name, aws colon region is passed to the actual region that this template is being applied into.

      So that's what the substitution function does.

      It replaces any variable or parameter names with the values of those variables or parameters.

      So the CFN init process is then able to consult the CloudFormation stack and retrieve the configuration information.

      That's all stored in the metadata section of this logical resource.

      Now I just want to draw your attention to this double hyphen config sets wordpress underscore install.

      This tells us what set of instructions we want CFN init to run.

      So if I just expand the metadata section here, we've got one or more config sets defined.

      In this case, we've only got the one which is wordpress underscore install.

      And this config set runs five individual items, one after the other.

      And these are called config keys.

      So install CFN, software install, configure instance, install wordpress and configure wordpress.

      Now these reference the config keys defined below.

      So you'll see that the same name install CFN, software install, configure instance, install wordpress and configure wordpress.

      You'll recognize a lot of the commands used because they're the same commands that install and configure wordpress.

      So in the software install config key, we're using the DNF package manager to install various software packages that we need for this installation, such as WGet, MariaDB, the Apache web server and various other utilities.

      Then another part is services and we're specifying that we want these services to be enabled and to be running.

      So this means that the service will be set to start up on instance boot and it will make sure that it's running right now.

      The next config key is configure instance.

      The files component of this can create files with a certain content.

      So we're creating a file called etc update-motd.d/40-cow.

      This is the part that we had to do manually before and this is the thing that adds the cow say banner.

      Then we're running some more procedural commands to set the database root password and to update this banner.

      Then we've got install wordpress, which uses a sources option to expand whatever is specified here into this directory.

      So this automatically handles the download and the unjzip and untarring of this archive into this folder and it can even do that with authentication if needed.

      We're creating another file this time to perform the configuration of wordpress and another file this time to create the database for wordpress.

      Then finally we've got the configure wordpress which fixes up the permissions and creates these databases.

      So this is doing the same thing as the procedural example in the previous demo.

      Instead of running all of these commands one by one, this is just using desired state.

      Now there is one more thing that I wanted to point out right at the top.

      This is the part that configures CFN init to keep watching the logical resource configuration inside the cloud formation stack.

      And if it notices that the metadata for EC2 instance inside the stack changes, then it will run CFN init again.

      Remember how in the theory lesson I mentioned that this process could cope with stack updates.

      So it doesn't only run once like user data does.

      Well, this is how it does that.

      This configures this automatic update that keeps an eye on the cloud formation stack and reruns CFN init whenever any changes occur.

      This is well beyond what you need for the associate exam.

      I just want you to be aware of what this is and how it works.

      Essentially we're setting up a process called CFN hop and making it watch the cloud formation stack for any configuration changes.

      And then we're setting it up so that the CFN hop process is enabled and running so that it can watch the resource configuration constantly.

      So that's it for this template.

      What we'll do now is apply it.

      So go ahead and click on the second one click deployment link attached to this lesson.

      It should be called A4LEC2CFN init.

      So click that link.

      All you'll need to do is scroll down to the bottom and then click on create stack.

      Now this time remember we're using a creation policy.

      So cloud formation is not going to move this logical ID and to create complete when EC2 signals that the launch process is completed.

      Instead it's going to wait until the instance itself signals the successful completion of the CFN init process.

      So because we're using this creation policy it's going to hold until the instance operating system using CFN-signal provide a signal to cloud formation to say yep everything's okay and at that point the logical resource will move into create complete.

      So that's going to take a couple of minutes.

      The EC2 instance will need to actually launch itself and pass its status checks and then the CFN init process will run, perform all of the configuration required and then assuming the status code of that is okay then CFN-signal will take that status code and respond to the cloud formation stack with a successful completion and then the process will move on then cloud formation will mark the particular resources complete and the stack is complete.

      Now that will take a few minutes so just keep hitting refresh and you should see the status update after two to three minutes but go ahead and pause the video and resume it once your stack moves into the create complete status.

      And there we go at this point the stack has moved into the create complete status and I just want to draw your attention to this line.

      You won't have seen this before.

      This is the line where our EC2 instance has run the CFN init process successfully and then the CFN signal command has taken that success signal and delivered it to the cloud formation stack.

      So this is the signal that cloud formation was waiting for before moving this resource into a create complete status and that's what's needed before the stack itself could move into a create complete status.

      So now we explicitly know that the configuration of this instance has actually been completed.

      So we're not relying on EC2 telling us that the instance status is now running with two out of two checks.

      Instead the operating system itself the CFN init process that's completed successfully and the CFN signal process has explicitly indicated to cloud formation that that whole process has been completed.

      So if we move across to the EC2 console we should be able to connect to the instance exactly as we've done before.

      Look for the running instance and select it.

      Copy the public IP version 4 IP address and open that in a new tab.

      All being well you should see the familiar WordPress installation screen.

      If you're right click on that instance and put connect.

      Go to instance connect and hit connect that will connect you into the instance and you should be greeted by the cow themed login banner.

      This time if we use curl to show us the contents of user data this time it's only a small number of lines because the only thing that runs is the CFN init process and the CFN signal process.

      Notice though how all of these variable names have been replaced with their values so the stack IDs and the region.

      So this is how it knows to communicate with the right stack in the right region inside cloud formation.

      If we do a CD space forward slash var forward slash log and then do a listing we've still got these original two files so cloud hyphen init dot log and cloud hyphen init hyphen output dot log.

      So these are primarily associated with the user data output.

      But now we've also got these new log files so CFN hyphen init hyphen CMD dot log and that is an output of the CFN init process.

      So if we cat that so shudu space cat space and then the name of that log file this will show us an output of the CFN init process itself.

      So we can see each of the individual config keys running and what individual operations are being performed inside each of those keys.

      So it's a more complex but a more powerful process.

      And at this point that's everything I wanted to cover.

      It was just to give you practical exposure to an alternative to raw user data and that was CFN hyphen init.

      It's a much more powerful system especially when combined with cloud formation creation policies which allow us to pause the progress of a cloud formation stack waiting for the resource itself to explicitly say yes I finished off all of my bootstrapping process you're good to carry on and that's done using the CFN hyphen signal command.

      Now at this point let's just clean up the account move back to cloud formation.

      Once you there go ahead and delete the EC2 CFN init stack wait for that process to complete and once you've done that go ahead and delete the A4L VPC stack and that will return the AWS account into the state that you had it at the start of this demo.

      At that point thanks for doing this demo I hope it was useful.

      You can go ahead and complete this video now and when you're ready you can join me in the next.

  3. Oct 2024
    1. Welcome back and in this demo lesson you're going to get the experience of bootstrapping an EC2 instance using user data.

      So this is the ability to run a script during the provisioning process for an EC2 instance and automatically add a certain configuration to that instance during the build process.

      So this is an alternative to creating a custom AMI.

      Earlier in the course you created an Amazon machine image with the WordPress installation and configuration baked in.

      Now that's really quick and simple but it does limit your ability to make changes to that configuration.

      So the configuration is baked into the AMI and so you're limited as to what you can change during launch time.

      With boot strapping you have the ability to perform all the steps in the form of a script during the provisioning process and so it can be a lot more flexible.

      Now to get started we need to create the Animals for Life VPC within our general AWS account.

      So this is the management account of the organization.

      So make sure that you're logged into the IAM admin user of this account and as always make sure you have the Northern Virginia region selected.

      Now attached to this lesson is a one-click deployment link so go ahead and open that.

      This is going to take you to the quick create stack page and everything should be pre-populated.

      The stack name should be bootstrap everything else has appropriate default so just scroll down to the bottom, check the capabilities acknowledgement box and then go ahead and click on create stack.

      Now this will create the Animals for Life VPC which contains the public subnets that we'll be launching our instance into and so we're going to need this to be in a create complete state before we move on.

      So go ahead and pause the video and once your stack changes from create in progress to create complete then we good to continue.

      Okay so now that that stack has moved into a create complete state we good to continue.

      Now also attached to this lesson is another link which is the user data that we're going to use for this demo lesson so go ahead and open that link.

      This is the user data that we're going to use to bootstrap the EC2 instance so what I want you to do is to download this file to your local machine and then open it in a code editor or alternatively just copy all the text on screen now and paste that into a code editor.

      So I've gone ahead and opened that file in my text editor and if you look through all of the different commands contained within this user data .txt file then you should recognize some of them.

      These are basically the commands that we ran earlier in the course when we manually installed word press and when we created the Amazon machine image.

      So we're essentially installing the MariaDB database server, the Apache web server, Wget and Cowsay.

      We're installing PHP and its associated libraries.

      We're making sure that both the database and the web server are set to automatically start when the instance reboots and are explicitly started when this script is run.

      We're setting the root password of the MariaDB database server.

      We're downloading the latest copy of the WordPress installation archive.

      We're extracting it and we're moving the files into the correct locations.

      Then we're configuring WordPress by copying the sample configuration file into the final and proper file name so wp-config.php and then we're performing a search and replace on those placeholders and replacing them with our actual chosen values for the database name, the database user and the database password.

      And then after that we're fixing up the permissions on the web root folder with the WordPress installation files inside so we're making sure that the ownership is correct and then we're fixing up the permissions with a slightly improved version of what we've used previously.

      Then we're creating our DB.setup script in the same way that we did when we were manually installing WordPress.

      We're logging into the database using the MySQL command line utility, authenticating as the root user with the root password and then running this script and this creates the WordPress database, the user sets the password and gives that user permissions on the database.

      And then finally we're configuring the Cowsay utility so we're setting up the message of the day file we're outputting our animals for life custom greeting and then we're forcing a refresh of the login banner.

      So these are all of the steps that you've previously done manually so I hope it's still fresh in your memory just how annoying that manual installation was.

      Okay so at this point this user data is ready to go and I want to demonstrate to you how you can use this to bootstrap an EC2 instance.

      So let's go ahead and move back to the AWS console.

      Once we're at the AWS console this CloudFormation 1 click deployment has created the Animals for Life VPC.

      So what we're going to do is to click on the services drop down and then move to the EC2 console and go ahead and click on launch instance followed by launch instance again.

      So first things first the instance is going to be called a4l for animals for life - manual WordPress so go ahead and enter that in the box at the top then scroll down select Amazon Linux and then make sure Amazon Linux 2023 is selected in the drop down and then make sure that you've got 64-bit x86 selected.

      I want you to pick whichever type is free tier eligible within your account and region in my case it's t2.micro but you should pick the one that's free tier eligible.

      Under key pair go ahead and pick proceed without a key pair then scroll down to network settings and click on edit and there are a few items on this page that we need to explicitly configure.

      The first is we need to select the Animals for Life VPC next to network so select a4l -vpc1 next to subnet I want you to go ahead and pick sn -web -a so that's the web or public subnet within availability zone a then make sure auto assign public IP is set to enable we'll be using an existing security group so check that box and then in the drop down so click the drop down and select the bootstrap -instance security group so bootstrap was the name of the cloud formation stack that we created using the one-click deployment we won't be making any changes to the storage configuration and next we need to scroll down to an option that we've not used before we're going to enter some user data so scroll all the way down and under advanced details expand this if it isn't already and you're looking for the user data box what we're going to do is paste in the user data that you just downloaded so in my case this is the user data.txt which I downloaded so I'm going to go ahead and select all of the information in this user data.txt making sure I get everything including the last line and I'm going to copy that into my clipboard now back at the AWS console we need to paste that in to the user data box now by default EC2 accepts user data as base64 encoded data so we need to provide it with base64 encoded data and we're not we're just giving it a normal text file so in this case the user interface can actually do this conversion for us so if what you're pasting in is not base64 encoded and what we're pasting in isn't then we don't need to do anything else if we're pasting in data which is already base64 encoded we need to check this box below the user data box we don't need to worry about that because we're not pasting in anything with base64 encoding so we can just paste in our user data directly into this box and this will be run during the instance launch process so this is where our automatic configuration comes from this is what will bootstrap the EC2 instance okay so that's everything we need to configure so go ahead and click on launch instance now at this point while this is launching I want you to keep in mind that in the previous demo examples in this course we manually launched an instance and then once the instance was in a running state we had to connect into it download WordPress install WordPress and then configure WordPress along with all of the other associated dependencies that WordPress requires so that was a fairly time-intensive process that was open to errors in the AMI example we followed that same process but at the end we created the Amazon machine image so keep that in mind and compare it to what your experience is in this demo lesson so now we've launched the instance and it's now in a running state and we've provided some user data to this instance so I want you to leave it a couple of minutes after it's showing in a running state just give it a brief while to perform that additional configuration after a few minutes go ahead and right click on that instance and select connect we're going to be using EC2 instance connect so make sure that's selected make sure the user is set to EC2 - user and then just click connect now what you should see if we've given this enough time is our custom animals for life login banner and that means that the bootstrapping process has completed think about this for a minute as part of the launch process EC2 has provisioned us an EC2 instance and it's also run a relatively complex installation and configuration script that we've supplied in the form of user data and that's downloaded and installed WordPress and configured our custom login banner if we go back to EC2 select instances and then if we copy the public IP address into our clipboard so copy the actual IP address do not click on this link because this will open it using HTTPS which we haven't configured if you take that IP address and open that in a new tab you'll see the installation dialogue for WordPress and that's because the bootstrapping process using the user data has done all the configuration process that previously we've had to do manually now if we go back to the instance I want to demonstrate architecturally and operationally exactly how this works what we can do is use the curl utility to review the instance metadata now because we're using Amazon Linux 2023 we need to do this slightly differently we need to use version 2 of the metadata service so first we need to run this command to get a token which we can use to authenticate to the metadata service so run this next we can run this command which gets us the metadata of the instance and this uses the 169254 169254 address or as I like to call it 169.254 repeating now if we use this with meta hyphen data on the end then we get the metadata service but as we know user data is a component of the metadata service so instead of using forward slash latest forward slash metadata we can replace metadata with user data and this will allow us to see the user data supplied to the instance and don't worry all of these commands will be attached to the lesson so you should recognize this this is the user data that we passed into the instance so this is performed a download a configuration and an installation of Apache the database server and WordPress as well as our custom login banner so that's how the user data gets into the EC2 instance and there's a service running on the EC2 instance which takes this data and automatically performs these configuration steps essentially this is run as a script on the operating system now something else we can do is to move into the forward slash VAR forward slash log folder and this is a folder which contains many of the system logs and if we do an LS space hyphen LA we'll see a collection of logs within this folder there are two logs in particular that are really useful for diagnosing bootstrapping related problems these logs are cloud hyphen init dot log and cloud hyphen init hyphen output dot log and both of these are used for slightly different reasons so what I want to do is to output one of these logs and show you the content so we're going to output using shudu first to get admin permissions and then cat and we're going to use the cloud hyphen init hyphen output dot log and I'm going to press enter and that's going to show you the contents of this file and you'll be able to see using this log file exactly what's been executed on this EC2 instance so you'll be able to see all of the actual commands and the output from those commands as they've been executed on this EC2 instance so you'll be able to see all of the WordPress related downloads and copies the replacements of the database usernames and passwords the permissions fix section the database creation user creation and then permissions on that database as well as the command that actually executes those and then right at the bottom is where we configure our custom login banner so this is how you can see exactly what's been run on this EC2 instance and if you ever encounter any issues with any of the demo lessons within this course or any of my courses then you can use this file to determine exactly what's happened on the EC2 instance as part of the bootstrapping process okay so this is the end of part one of this lesson it was getting a little bit on the long side and so I wanted to add a break it's an opportunity just to take a rest or grab a coffee part two will be continuing immediately from the end of part one so go ahead complete the video and when you're ready join me in part two.

    1. Welcome back and in this demo lesson you're going to be creating an ECS cluster with the Fargate cluster mode and using the container of CATS container that we created together earlier in this section of the course, you're going to deploy this container into your Fargate cluster.

      So you're going to get some practical experience of how to deploy a real container into a Fargate cluster.

      Now you won't need any cloud formation templates applied to perform this demo because we're going to use the default VPC.

      All that you'll need is to be logged in as the IAM admin user inside the management account of the organization and just make sure that you're in the Northern Virginia region.

      Once you've confirmed that then just click in Find Services and type ECS and then click to move to the ECS console.

      Once you're at the ECS console, step one is to create a Fargate cluster.

      So that's the cluster that our container is going to run inside.

      So click on clusters, then create cluster.

      You'll need to give the cluster a name.

      You can put anything you want here, but I recommend using the same as me and I'll be putting all the CATS.

      Now Fargate mode requires a VPC.

      I'm going to be suggesting that we use the default VPC because that's already configured, remember, to give public IP addresses to anything deployed into the public subnets.

      So just to keep it simple and avoid any extra configuration, we'll use the default VPC.

      Now it should automatically select all of the subnets within the default VPC, in my case all six.

      If yours doesn't, just make sure you select all of the available subnets from this dropdown, but it should do this by default.

      Then scroll down and just note how AWS Fargate is already selected and that's the default.

      If you wanted to, you could check to use Amazon EC2 instances or external instances using ECS anywhere, but for this demo, we won't be doing that.

      Instead, we'll leave everything else as default, scroll down to the bottom and click create.

      If this is the first time you're doing this in an AWS account, it's possible that you'll get the error that's shown on screen now.

      If you do get this error, then what I would suggest is to wait a few minutes, then go back to the main ECS console, go to cluster again and then create the all the cats cluster again.

      So follow exactly the same steps, call the cluster all the cats, make sure that the animals for live default VPC is selected and all those subnets are present, and then click on create.

      You should find that the second time that you run this creation process, it works okay.

      Now this generally happens because there's an approval process that needs to happen behind the scenes.

      So if this is the first time that you're using ECS within this AWS account, then you might get this error.

      It's nothing to worry about, just rerun the process and it should create fine the second time.

      Once you've followed that process through again, or if it works the first time, then just go ahead and click on the all the cats cluster.

      So this is the Fargate based cluster.

      It's in an active state, so we're good to deploy things into this cluster.

      And we can see that we've got no active services.

      If I click on tasks, we can see we've got no active tasks.

      There's a tab here, metrics where you can see cloud watch metrics about this cluster.

      And again, because this is newly created and it doesn't have any activity, all of this is going to be blank.

      For now, that's fine.

      What we need to do for this demonstration is create a task definition that will deploy our container, our container of cats container into this Fargate cluster.

      To do that, click on task definitions and create a new task definition.

      You'll need to pick a name for your task definition.

      Go ahead and put container of cats.

      And then inside this task definition, the first thing to do set the details of the container for this task.

      So under container details under name, go ahead and put container of cats web.

      So this is going to be the web container for the container of cats task.

      Then next to the name under image URI, you need to point this at the docker image that's going to be used for this container.

      So I'm going to go ahead and paste in the URI for my docker image.

      So this is the docker image that I created earlier in the course within the EC2 docker demo.

      You might have also created your own container image.

      You can feel free to use my container image or you can use yours.

      If you want to keep things simple, you should go ahead and use mine.

      Yours should be the same anyway.

      Now just to be careful, this isn't a URL.

      This is a URI to point at my docker image.

      So it consists of three parts.

      First we have docker.io, which is the docker hub.

      Then we have my username, so acantral.

      And then we have the repository name, which is container of cats.

      So if you want to use your own docker image, you need to change both the username and the repository name.

      Again, to keep things simple, feel free to use my docker image.

      Then scrolling down, we need to make sure that the port mappings are correct.

      It should show what's on screen now, so container port 80, TCP.

      And then the port name should be the same or similar to what's on screen now.

      Don't worry if it's slightly different and the application protocol should be HTTP.

      This is controlling the port mapping from the container through to the Fargate IP address.

      And I'll talk more about this IP address later on in this demo.

      Everything else looks good, so scroll down to the bottom and click on next.

      We need to specify some environment details.

      So under operating system/architecture, it needs to be linux/x86_64.

      Under task size for memory, go ahead and select 1GB and then under CPU, 0.5 vCPU.

      That should be enough resources for this simple docker application.

      Scroll down and under monitoring and logging, uncheck use log collection.

      We won't be needing it for this demo lesson.

      That's everything we need to do.

      Go ahead and click on next.

      This is just an overview of everything that we've configured, so you can scroll down to the bottom and click on create.

      And at this point, the task definition has been created successfully.

      And this is where you can see all of the details of the task definition.

      If you want to see the raw JSON for the task definition itself, you don't need this for the exam, but this is actually what a task definition looks like.

      So it contains all of this different information.

      What it has got is one or more container definitions.

      So this is just JSON.

      This is a list of container definitions.

      We've only got the one.

      And if you're looking at this, you can see where we set the port mapping.

      So we're mapping port 80.

      You can see where it's got the image URL, which is where it pulls the docker image from.

      This is exactly what a normal task and container definition look like.

      They can be significantly more complex, but this format is consistent across all task definitions.

      Okay, so now it's time to launch a task.

      It's time to take the container and task definitions that we've defined and actually run up a container inside ECS using those definitions.

      So to do that, click on clusters and then select the all the cats cluster.

      Click on tasks and then click on run a new task.

      Now, first we need to pick the compute options and we're going to select launch type.

      So check that box.

      If appropriate for the certification that you're studying for, I'll be talking about the differences between these two in a different lesson.

      Once you've clicked on launch type, make sure Fargate is selected in the launch type drop down and latest is selected under platform version.

      Then scroll down and we're going to be creating a task.

      So make sure that task is selected.

      Scroll down again and under family, make sure container of cats is selected.

      And then under revision, select latest.

      We want to make sure the latest version is used and we'll leave desired tasks at one and task group blank.

      Scroll down and expand networking.

      Make sure the default VPC is selected and then make sure again that all of the subnets inside the default VPC are present under subnets.

      The default is that all of them should be in my case six.

      Now the way that this task is going to work is that when the task is run within Fargate, an elastic network interface is going to be created within the default VPC.

      And that elastic network interface is going to have a security group.

      So we need to make sure that the security group is appropriate and allows us to access our containerized application.

      So check the box to say create a new security group and then for security group name and description, use container of cats -sg.

      We need to make sure that the rule on this security group is appropriate.

      So under type select HTTP and then under source change this to anywhere.

      And this will mean that anyone can access this containerized application.

      Finally make sure that public IP is turned on.

      This is really important because this is how we'll access our containerized application.

      Everything else looks good.

      We can scroll down to the bottom and click on create.

      Now give that a couple of seconds.

      It should initially show last status.

      So the last status should be set to provisioning and the desired state should be set to running.

      So we need to wait for this task provisioning to complete.

      So just keep hitting refresh.

      You'll see it first change into pending.

      Now at this point we need this task to be in a running state before we can continue.

      So go ahead and pause the video and wait for both of these states.

      So last status and desired status both of those need to be running before we continue.

      So pause the video, wait for both of those to change and then once they have you can resume and will continue.

      After another refresh the last status should now be running and in green and the desired state should also be running.

      So at that point we're good to go.

      We can click on the task link below.

      We can scroll down and our task has been allocated a private IP version for address in the default VPC and also a public IP version for address also in the default VPC.

      So if we copy this public IP into our clipboard and then open a new tab and browse to this IP we'll see our very corporate professional web application.

      If it fits, I sits in a container in a container.

      So we've taken a Docker image that we created earlier in this section of the course.

      We've created a Fargate cluster, created a task definition with a container definition inside and deployed our container image as a container to this Fargate cluster.

      So it's a very simple example, but again this scales.

      So you could deploy Docker containers which are a lot more complex in what functionality they offer.

      In this case it's just an Apache web server loading up a web page but we could deploy any type of web application using the same steps that you've performed in this demo lesson.

      So congratulations, you've learned all of the theory that you'll need for the exam and you've taken the steps to implement this theory in practice by deploying a Docker image as a container on an ECS Fargate cluster.

      So great job.

      At this point all that remains is to tidy up.

      So go back to the AWS console.

      Just stop this container.

      Click on stop.

      Click on task definitions and then go into this task definition.

      Select this.

      Click on actions, deregister and then click on deregister.

      Click back on task definitions and make sure there's no results there.

      That's good.

      Click on clusters.

      Click on all the cats.

      Delete the cluster.

      You'll need to type delete space all the cats and then click on delete to confirm.

      And at that point the Fargate cluster has been deleted.

      The running container has been stopped.

      The task definitions been deleted and our account is back in the same state as when we started.

      So at this point you've completed the demo.

      You've done great and you've implemented some pretty complex theory.

      So you should already have a head start on any exam questions which involve ECS.

      We're going to be using ECS a lot more as we move through the course and we're going to be using it in some of the Animals for Life demos as we implement progressively more complex architectures later on in the course.

      For now I just wanted to give you the basics but you've done really well if you've implemented this successfully without any issues.

      So at this point go ahead, complete this video and when you're ready join me in the next.

    1. Welcome back and in this demo lesson you're going to learn how to install the Docker engine inside an EC2 instance and then use that to create a Docker image.

      Now this Docker image is going to be running a simple application and we'll be using this Docker image later in this section of the course to demonstrate the Elastic Container service.

      So this is going to be a really useful demo where you're going to gain the experience of how to create a Docker image.

      Now there are a few things that you need to do before we get started.

      First as always make sure that you're logged in to the I am admin user of the general AWS account and you'll also need the Northern Virginia region selected.

      Now attached to this lesson is a one-click deployment link so go ahead and click that now.

      This is going to deploy an EC2 instance with some files pre downloaded that you'll use during the demo lesson.

      Now everything's pre-configured you just need to check this box at the bottom and click on create stack.

      Now that's going to take a few minutes to create and we need this to be in a create complete state.

      So go ahead and pause the video wait for your stack to move into create complete and then we're good to continue.

      So now this stack is in a create complete state and we're good to continue.

      Now if you're following along with this demo within your own environment there's another link attached to this lesson called the lesson commands document and that will include all of the commands that you'll need to type as you move through the demo.

      Now I'm a fan of typing all commands in manually because I personally think that it helps you learn but if you are the type of person who has a habit of making mistakes when typing along commands out then you can copy and paste from this document to avoid any typos.

      Now one final thing before we finish at the end of this demo lesson you'll have the opportunity to upload the Docker image that you create to Docker Hub.

      If you're going to do that then you should pre sign up for a Docker Hub account if you don't already have one and the link for this is included attached to this lesson.

      If you already have a Docker Hub account then you're good to continue.

      Now at this point what we need to do is to click on the resources tab of this stack and locate the public EC2 resource.

      Now this is a normal EC2 instance that's been provisioned on your behalf and it has some files which have been pre downloaded to it.

      So just go ahead and click on the physical ID next to public EC2 and that will move you to the EC2 console.

      Now this machine is set up and ready to connect to and I've configured it so that we can connect to it using Session Manager and this avoids the need to use SSH keys.

      So to do that just right-click and then select connect.

      You need to pick Session Manager from the tabs across the top here and then just click on connect.

      Now that will take a few minutes but once connected you should see this prompt.

      So it should say SH- and then a version number and then dollar.

      Now the first thing that we need to do as part of this demo lesson is to install the Docker engine.

      The Docker engine is the thing that allows Docker containers to run on this EC2 instance.

      So we need to install the Docker engine package and we'll do that using this command.

      So we're using shudu to get admin permissions then the package manager DNF then install then Docker.

      So go ahead and run that and that will begin the installation of Docker.

      It might take a few moments to complete it might have to download some prerequisites and you might have to answer that you're okay with the install.

      So press Y for yes and then press enter.

      Now we need to wait a few moments for this install process to complete and once it has completed then we need to start the Docker service and we do that using this command.

      So shudu again to get admin permissions and then service and then the Docker service and then start.

      So type that and press enter and that starts the Docker service.

      Now I'm going to type clear and then press enter to make this easier to see and now we need to test that we can interact with the Docker engine.

      So the most simple way to do that is to type Docker space and then PS and press enter.

      Now you're going to get an error.

      This error is because not every user of this EC2 instance has the permissions to interact with the Docker engine.

      We need to grant permissions for this user or any other users of this EC2 instance to be able to interact with the Docker engine and we're going to do that by adding these users to a group and we do that using this command.

      So shudu for admin permissions and then user mod -a -g for group and then the Docker group and then EC2 -user.

      Now that will allow a local user of this system, specifically EC2 -user, to be able to interact with the Docker engine.

      Okay so I've cleared the screen to make it slightly easier to see now that we've added EC2 -user the ability to interact with Docker.

      So the next thing is we need to log out and log back in of this instance.

      So I'm going to go ahead and type exit just to disconnect from session manager and then click on close and then I'm going to reconnect to this instance and you need to do the same.

      So connect back in to this EC2 instance.

      Now once you're connected back into this EC2 instance we need to run another command which moves us into EC2 user so it basically logs us in as EC2 -user.

      So that's this command and the result of this would be the same as if you directly logged in to EC2 -user.

      Now the reason we're doing it this way is because we're using session manager so that we don't need a local SSH client or to worry about SSH keys.

      We can directly log in via the console UI we just then need to switch to EC2 -user.

      So run this command and press enter and we're now logged into the instance using EC2 -user and to test everything's okay we need to use a command with the Docker engine and that command is Docker space ps and if everything's okay you shouldn't see any output beyond this list of headers.

      What we've essentially done is told the Docker engine to give us a list of any running containers and even though we don't have any it's not erred it's simply displayed this empty list and that means everything's okay.

      So good job.

      Now what I've done to speed things up if you just run an LS and press enter the instance has been configured to download the sample application that we're going to be using and that's what the file container.zip is within this folder.

      I've configured the instance to automatically extract that zip file which has created the folder container.

      So at this point I want you to go ahead and type cd space container and press enter and that's going to move you inside this container folder.

      Then I want you to clear the screen by typing clear and press enter and then type ls space -l and press enter.

      Now this is the web application which I've configured to be automatically downloaded to the EC2 instance.

      It's a simple web page we've got index.html which is the index we have a number of images which this index.html contains and then we have a docker file.

      Now this docker file is the thing that the docker engine will use to create our docker image.

      I want to spend a couple of moments just stepping you through exactly what's within this docker file.

      So I'm going to move across to my text editor and this is the docker file that's been automatically downloaded to your EC2 instance.

      Each of these lines is a directive to the docker engine to perform a specific task and remember we're using this to create a docker image.

      This first line tells the docker engine that we want to use version 8 of the Red Hat Universal base image as the base component for our docker image.

      This next line sets the maintainer label it's essentially a brief description of what the image is and who's maintaining it in this case it's just a placeholder of animals for life.

      This next line runs a command specifically the yum command to install some software specifically the Apache web server.

      This next command copy copies files from the local directory when you use the docker command to create an image so it's copying that index.html file from this local folder that I've just been talking about and it's going to put it inside the docker image in this path so it's going to copy index.html to /var/www/html and this is where an Apache web server expects this index.html to be located.

      This next command is going to do the same process for all of the jpegs in this folder so we've got a total of six jpegs and they're going to be copied into this folder inside the docker image.

      This line sets the entry point and this essentially determines what is first run when this docker image is used to create a docker container.

      In this example it's going to run the Apache web server and finally this expose command can be used for a docker image to tell the docker engine which services should be exposed.

      Now this doesn't actually perform any configuration it simply tells the docker engine what port is exposed in this case port 80 which is HTTP.

      Now this docker file is going to be used when we run the next command which is to create a docker image.

      So essentially this file is the same docker file that's been downloaded to your EC2 instance and that's what we're going to run next.

      So this is the next command within the lesson commands document and this command builds a container image.

      What we're essentially doing is giving it the location of the docker file.

      This dot at the end contains the working directory so it's here where we're going to find the docker file and any associated files that that docker file uses.

      So we're going to run this command and this is going to create our docker image.

      So let's go ahead and run this command.

      It's going to download version 8 of UBI which it will use as a starting point and then it's going to run through every line in the docker file performing each of the directives and each of those directives is going to create another layer within the docker image.

      Remember from the theory lesson each line within the docker file generally creates a new file system layer so a new layer of a docker image and that's how docker images are efficient because you can reuse those layers.

      Now in this case this has been successful.

      We've successfully built a docker image with this ID so it's giving it a unique ID and it's tagged this docker image with this tag colon latest.

      So this means that we have a docker image that's now stored on this EC2 instance.

      Now I'll go ahead and clear the screen to make it easier to see and let's go ahead and run the next command which is within the lesson commands document and this is going to show us a list of images that are on this EC2 instance but we're going to filter based on the name container of cats and this will show us the docker image which we've just created.

      So the next thing that we need to do is to use the docker run command which is going to take the image that we've just created and use it to create a running container and it's that container that we're going to be able to interact with.

      So this is the command that we're going to use it's the next one within the lesson commands document.

      It's docker run and then it's telling it to map port 80 on the container with port 80 on the EC2 instance and it's telling it to use the container of cats image and if we run that command docker is going to take the docker image that we've got on this EC2 instance run it to create a running container and we should be able to interact with that container.

      So if you go back to the AWS console if we click on instances so look for a4l-public EC2 that's in the running state.

      I'm just going to go ahead and select this instance so that we can see the information and we need the public IP address of this instance.

      Go ahead and click on this icon to copy the public IP address into your clipboard and then open that in a new tab.

      Now be sure not to use this link to the right because that's got a tendency to open the HTTPS version.

      We just need to use the IP address directly.

      So copy that into your clipboard open a new tab and then open that IP address and now we can see the amazing application if it fits i sits in a container in a container and this amazing looking enterprise application is what's contained in the docker image that you just created and it's now running inside a container based off that image.

      So that's great everything's working as expected and that's running locally on the EC2 instance.

      Now in the demo lesson for the elastic container service that's coming up later in this section of the course you have two options.

      You can either use my docker image which is this image that I've just created or you can use your own docker image.

      If you're going to use my docker image then you can skip this next step.

      You don't need a docker hub account and you don't need to upload your image.

      If you want to use your own image then you do need to follow these next few steps and I need to follow them anyway because I need to upload this image to docker hub so that you can potentially use it rather than your own image.

      So I'm going to move back to the session manager tab and I'm going to control C to exit out of this running container and I'm going to type clear to clear the screen and make it easier to see.

      Now to upload this to docker hub first you need to log in to docker hub using your credentials and you can do that using this command.

      So it's docker space login space double hyphen username equals and then your username.

      So if you're doing this in your own environment you need to delete this placeholder and type your username.

      I'm going to type my username because I'll be uploading this image to my docker hub.

      So this is my docker hub username and then press enter and it's going to ask for the corresponding password to this username.

      So I'm going to paste in my password if you're logging into your docker hub you should use your password.

      Once you've pasted in the password go ahead and press enter and that will log you in to docker hub.

      Now you don't have to worry about the security message because whilst your docker hub password is going to be stored on the EC2 instance shortly we're going to terminate this instance which will remove all traces of this password from this machine.

      Okay so again we're going to upload our docker image to docker hub so let's run this command again and you'll see because we're just using the docker images command we can see the base image as well as our image.

      So we can see red hat UBI 8.

      We want the container of cats latest though so what you need to do is copy down the image ID of the container of cats image.

      So this is the top line in my case container of cats latest and then the image ID.

      So then we need to run this command so docker space tag and then the image ID that you've just copied into your clipboard and then a space and then your docker hub username.

      In my case it's actrl with 1L if you're following along you need to use your own username and then forward slash and then the name of the image that you want this to be stored as on docker hub so I'm going to use container of cats.

      So that's the command you need to use so docker tag and then your image ID for container of cats and then your username forward slash container of cats and press enter and that's everything we need to do to prepare to upload this image to docker hub.

      So the last command that we need to run is the command to actually upload the image to docker hub and that command is docker space push so we're going to push the image to docker hub then we need to specify the docker hub username so again this is my username but if you're doing this in your environment it needs to be your username and then forward slash and then the image name in my case container of cats and then colon latest and once you've got all that go ahead and press enter and that's going to push the docker image that you've just created up to your docker hub account and once it's up there it means that we can deploy from that docker image to other EC2 instances and even ECS and we're going to do that in a later demo in this section of the course.

      Now that's everything that you need to do in this demo lesson you've essentially installed and configured the docker engine you've used a docker file to create a docker image from some local assets you've tested that docker image by running a container using that image and then you've uploaded that image to docker hub and as I mentioned before we're going to use that in a future demo lesson in this section of the course.

      Now the only thing that remains to do is to clear up the infrastructure that we've used in this demo lesson so go ahead and close down all of these extra tabs and go back to the cloud formation console this is the stack that's been created by the one click deployment link so all you need to do is select this stack it should be called EC2 docker and then click on delete and confirm that deletion and that will return the account into the same state as it was at the start of this demo lesson.

      Now that is everything you need to do in this demo lesson I hope it's been useful and I hope you've enjoyed it so go ahead and complete the video and when you're ready I look forward to you joining me in the next.

    1. Welcome back and in this very brief demo lesson, I just want to demonstrate a very specific feature of EC2 known as termination protection.

      Now you don't have to follow along with this in your own environment, but if you are, you should still have the infrastructure created from the previous demo lesson.

      And also if you are following along, you need to be logged in as the I am admin user to the general AWS account.

      So the management account of the organization and have the Northern Virginia region selected.

      Now again, this is going to be very brief.

      So it's probably not worth doing in your own environment unless you really want to.

      Now what I want to demonstrate is termination protection.

      So I'm going to go ahead and move to the EC2 console where I still have an EC2 instance running created in the previous demo lesson.

      Now normally if I right click on this instance, I'm given the ability to stop the instance, to reboot the instance or to terminate the instance.

      And this is assuming that the instance is currently in a running state.

      Now if I go to terminate instance, straight away I'm presented with a dialogue where I need to confirm that I want to terminate this instance.

      But it's easy to imagine that somebody who's less experienced with AWS can go ahead and terminate that and then click on terminate to confirm the process without giving it much thought.

      And that can result in data loss, which isn't ideal.

      What you can do to add another layer of protection is to right click on the instance, go to instance settings, and then change termination protection.

      If you click that option, you get this dialogue where you can enable termination protection.

      So I'm going to do that, I'm going to enable termination protection because this is an essential website for animals for life.

      So I'm going to enable it and click on save.

      And now that instance is protected against termination.

      If I right click on this instance now and go to terminate instance and then click on terminate, I get a dialogue that I'm unable to terminate the instance.

      The instance and then the instance ID may not be terminated, modify its disable API termination instance attribute and then try again.

      So this instance is now protected against accidental termination.

      Now this presents a number of advantages.

      One, it protects against accidental termination, but it also adds a specific permission that is required in order to terminate an instance.

      So you need the permission to disable this termination protection in addition to the permissions to be able to terminate an instance.

      So you have the option of role separation.

      You can either require people to have both the permissions to disable termination protection and permissions to terminate, or you can give those permissions to separate groups of people.

      So you might have senior administrators who are the only ones allowed to remove this protection, and junior or normal administrators who have the ability to terminate instances, and that essentially establishes a process where a senior administrator is required to disable the protection before instances can be terminated.

      It adds another approval step to this process, and it can be really useful in environments which contain business critical EC2 instances.

      So you might not have this for development and test environments, but for anything in production, this might be a standard feature.

      If you're provisioning instances automatically using cloud formation or other forms of automation, this is something that you can enable in an automated way as instances are launching.

      So this is a really useful feature to be aware of.

      And for the SysOps exam, it's essential that you understand when and where you'd use this feature.

      And for both the SysOps and the developer exams, you should pay attention to this, disable API termination.

      You might be required to know which attribute needs to be modified in order to allow terminations.

      So really for both of the exams, just make sure that you're aware of exactly how this process works end to end, specifically the error message that you might get if this attribute is enabled and you attempt to terminate an instance.

      At this point though, that is everything that I wanted to cover about this feature.

      So right click on the instance, go to instance settings, change the termination protection and disable it, and then click on save.

      One other feature which I want to introduce quickly, if we right click on the instance, go to instance settings, and then change shutdown behavior, you're able to specify whether an instance should move into a stop state when shut down, or whether you want it to move into a terminate state.

      Now logically, the default is stop, but if you are running an environment where you don't want to consider the state of an instance to be valuable, then potentially you might want it to terminate when it shuts down.

      You might not want to have an account with lots of stopped instances.

      You might want the default behavior to be terminate, but this is a relatively niche feature, and in most cases, you do want the shutdown behavior to be stop rather than terminate, but it's here where you can change that default behavior.

      Now at this point, that is everything I wanted to cover.

      If you were following along with this in your own environment, you do need to clear up the infrastructure.

      So click on the services dropdown, move to cloud formation, select the status checks and protect stack, and then click on delete and confirm that by clicking delete stack.

      And once this stack finishes deleting all of the infrastructure that's been used during this demo and the previous one will be cleared from the AWS account.

      If you've just been watching, you don't need to worry about any of this process, but at this point, we're done with this demo lesson.

      So go ahead, complete the video, and once you're ready, I'll look forward to you joining me in the next.

    1. Welcome back and in this demo lesson either you're going to get the experience or you can watch me interacting with an Amazon machine image.

      So we created an Amazon machine image or AMI in a previous demo lesson and if you recall it was customized for animals for life.

      It had an install of WordPress and it had the Kause application installed and a custom login banner.

      Now this is a really simple example of an AMI but I want to step you through some of the options that you have when dealing with AMIs.

      So if we go to the EC2 console and if you are following along with this in your own environment do make sure that you're logged in as the IAM admin user of the general AWS account, so the management account of the organization and you have the Northern Virginia region selected.

      The reason for being so specific about the region is that AMIs are regional entities so you create an AMI in a particular region.

      So if I go and select AMIs under images within the EC2 console I'll see the animals for life AMI that I created in a previous demo lesson.

      Now if I go ahead and change the region maybe from Northern Virginia which is US-East-1 to US-East- Ohio which is US-East-2 if I make that change what we'll see is we'll go back to the same area of the console only now we won't see any AMIs that's because an AMI is tied to the region in which it's created.

      Every AMI belongs in one region and it has a unique AMI ID.

      So let's move back to Northern Virginia.

      Now we are able to copy AMIs between regions this allows us to make one AMI and use it for a global infrastructure platform so we can right-click and select copy AMI then select the destination region and then for this example let's say that I did want to copy it to Ohio then I would select that in the drop-down it would allow me to change the name if I wanted or I could keep it the same for description it would show that it's been copied from this AMI ID in this region and then it would have the existing description at the end.

      So at this point I'm going to go ahead and click copy AMI and that process has now started so if I close down this dialogue and then change it from US East 1 to US East 2 so select that now we have a pending AMI and this is the AMI that's being copied from the US - East - one region into this region if we go ahead and click on snapshots under elastic block store then we're going to see the snapshot or snapshots which belong to this AMI.

      Now depending on how busy AWS is it can take a few minutes for the snapshots to appear on this screen just go ahead and keep refreshing until they appear.

      In our case we only have the one which is the boot volume that's used for our custom AMI.

      Now the time taken to copy a snapshot between regions depends on many factors what the source and destination region are and the distance between the two the size of the snapshot and the amount of data it contains and it can take anywhere from a few minutes to much much longer so this is not an immediate process.

      Once the snapshot copy completes then the AMI copy process will complete and that AMI is then available in the destination region but an important thing that I want to keep stressing throughout this course is that this copied AMI is a completely different AMI.

      AMIs are regional don't fall for any exam questions which attempt to have you use one AMI for several regions.

      If we're copying this animals for life AMI from one region to another region in effect we're creating two different AMIs.

      So take note of this AMI ID in this region and if we switch back to the original source region so US - East - 1 note how this AMI has a different ID so they are different AMIs completely different AMIs you're creating a new one as part of the copy process.

      So while the data is going to be the same conceptually they are completely separate objects and that's critical for you to understand both for production usage and when answering any exam questions.

      Now while that's copying I want to demonstrate the other important thing which I wanted to show you in this demo lesson and that's permissions of AMIs.

      So if I right-click on this AMI and edit AMI permissions by default an AMI is private.

      Being private means that it's only accessible within the AWS account which has created the AMI and so only identities within that account that you grant permissions are able to access it and use it.

      Now you can change the permission of the AMI you could set it to be public and if you set it to public it means that any AWS account can access this AMI and so you need to be really careful if you select this option because you don't want any sensitive information contained in that snapshot to be leaked to external AWS accounts.

      A much safer way is if you do want to share the AMI with anyone else then you can select private but explicitly add other AWS accounts to be able to interact with this AMI.

      So I could click in this box and then for example if I clicked on services and I just moved to the AWS organization service I'll open that in a new tab and let's say that I chose to share this AMI with my production account so I selected my production account ID and then I could add this into this box which would grant my production AWS account the ability to access this AMI.

      Now no tell there's also this checkbox and this adds create volume permissions to the snapshots associated with this AMI so this is something that you need to keep in mind.

      Generally if you are sharing an AMI to another account inside your organization then you can afford to be relatively liberal with permissions so generally if you're sharing this internally I would definitely check this box and that gives full permissions on the AMI as well as the snapshots so that anyone can create volumes from those snapshots as well as accessing the AMI.

      So these are all things that you need to consider.

      Generally it's much preferred to explicitly grant an AWS account permissions on an AMI rather than making that AMI public.

      If you do make it public you need to be really sure that you haven't leaked any sensitive information, specifically access keys.

      While you do need to be careful of that as well if you're explicitly sharing it with accounts, generally if you're sharing it with accounts then you're going to be sharing it with trusted entities.

      You need to be very very careful if ever you're using this public option and I'll make sure I include a link attached to this lesson which steps through all of the best practice steps that you need to follow if you're sharing an AMI publicly.

      There are a number of really common steps that you can use to minimize lots of common security issues and that's something you should definitely do if you're sharing an AMI.

      Now if you want to do you could also share an AMI with an organizational unit or organization and you can do that using this option.

      This makes it easier if you want to share an AMI with all AWS accounts within your organization.

      At this point though I'm not going to do that we don't need to do that in this demo.

      What we're going to do now though is move back to US-East-2.

      That's everything I wanted to cover in this demo lesson.

      Now this AMI is available we can right click and select D register and move back to US-East-1 and now that we've done this demo lesson we can do the same process with this AMI.

      So we can right click select D register and that will remove that AMI.

      Click on snapshots this is the snapshot created by this AMI so we need to delete this as well right click delete that snapshot confirm that and we'll need to do the same process in the region that we copied the AMI and the snapshots to.

      So select US-East-2 it should be the only snapshot in the region make sure it is the correct one right click delete confirm that deletion and now you've cleared up all of the extra things created within this demo lesson.

      Now that's everything that I wanted to cover I just wanted to give you an overview of how to work with AMIs from the console UI from a copying and sharing perspective.

      Go ahead and complete this video and when you're ready I look forward to you joining me in the next.

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      So the first step is to shut down this instance.

      So we don't want to create an AMI from a running instance because that can cause consistency issues.

      So we're going to close down this tab.

      We're going to return to instances, right-click, and we're going to stop the instance.

      We need to acknowledge this and then we need to wait for the instance to change into the stopped state.

      It will start with stopping.

      We'll need to refresh it a few times.

      There we can see it's now in a stopped state and to create the AMI, we need to right-click on that instance, go down to Image and Templates, and select Create Image.

      So this is going to create an AMI.

      And first we need to give the AMI a name.

      So let's go ahead and use Animals for Life template WordPress.

      And we'll use the same for Description.

      Now what this process is going to do is it's going to create a snapshot of any of the EBS volumes, which this instance is using.

      It's going to create a block device mapping, which maps those snapshots onto a particular device ID.

      And it's going to use the same device ID as this instance is using.

      So it's going to set up the storage in the same way.

      It's going to record that storage inside the AMI so that it's identical to the instance we're creating the AMI from.

      So you'll see here that it's using EBS.

      It's got the original device ID.

      The volume type is set to the same as the volume that our instance is using, and the size is set to 8.

      Now you can adjust the size during this process as well as being able to add volumes.

      But generally when you're creating an AMI, you're creating the AMI in the same configuration as this original instance.

      Now I don't recommend creating an AMI from a running instance because it can cause consistency issues.

      If you create an AMI from a running instance, it's possible that it will need to perform an instance reboot.

      You can force that not to occur, so create an AMI without rebooting.

      But again, that's even less ideal.

      The most optimal way for creating an AMI is to stop the instance and then create the AMI from that stopped instance, which will have fully consistent storage.

      So now that that's set, just scroll down to the bottom and go ahead and click on Create Image.

      Now that process will take some time.

      If we just scroll down, look under Elastic Block Store and click on Snapshots.

      You'll see that initially it's creating a snapshot of the boot volume of our original EC2 instance.

      So that's the first step.

      So in creating the AMI, what needs to happen is a snapshot of any of the EBS volumes attached to that EC2 instance.

      So that needs to complete first.

      Initially it's going to be an appending state.

      We'll need to give that a few moments to complete.

      If we move to AMIs, we'll see that the AMI is also creating it too.

      It is in appending state and it's waiting for that snapshot to complete.

      Now creating a snapshot is storing a full copy of any of the data on the original EBS volume.

      And the time taken to create a snapshot can vary.

      The initial snapshot always takes much longer because it has to take that full copy of data.

      And obviously depending on the size of the original volume and how much data is being used, will influence how long a snapshot takes to create.

      So the more data, the larger the volume, the longer the snapshot will take.

      After a few more refreshes, the snapshot moves into a completed status and if we move across to AMIs under images, after a few moments this too will change away from appending status.

      So let's just refresh it.

      After a few moments, the AMI is now also in an available state and we're good to be able to use this to launch additional EC2 instances.

      So just to summarize, we've launched the original EC2 instance, we've downloaded, installed and configured WordPress, configured that custom banner.

      We've shut down the EC2 instance and generated an AMI from that instance.

      And now we have this AMI in a state where we can use it to create additional instances.

      So we're going to do that.

      We're going to launch an additional instance using this AMI.

      While we're doing this, I want you to consider exactly how much quicker this process now is.

      So what I'm going to do is to launch an EC2 instance from this AMI and note that this instance will have all of the configuration that we had to do manually, automatically included.

      So right click on this AMI and select launch.

      Now this will step you through the launch process for an EC2 instance.

      You won't have to select an AMI because obviously you are now explicitly using the one that you've just created.

      You'll be asked to select all of the normal configuration options.

      So first let's put a name for this instance.

      So we'll use the name "instance" from AMI.

      Then we'll scroll down.

      As I mentioned moments ago, we don't have to specify an AMI because we're explicitly launching this instance from an AMI.

      Scroll down.

      You'll need to specify an instance type just as normal.

      We'll use a free tier eligible instance.

      This is likely to be T2 or T3.micro.

      Below that, go ahead and click and select Proceed without a key pair not recommended.

      Scroll down.

      We'll need to enter some networking settings.

      So click on Edit next to Network Settings.

      Click in VPC and select A4L-VPC1.

      Click in Subnet and make sure that SN-Web-A is selected.

      Make sure the box is below a both set to enable for the auto assign IP settings.

      Under Firewall, click on Select Existing Security Group.

      Click in the Security Groups drop down and select AMI-Demo-Instance Security Group.

      And that will have some random at the end.

      That's absolutely fine.

      Select that.

      Scroll down.

      And notice that the storage is configured exactly the same as the instance which you generated this AMI from.

      Everything else looks good.

      So we can go ahead and click on Launch Instance.

      So this is launching an instance using our custom created AMI.

      So let's close down this dialog and we'll see the instance initially in a pending state.

      Remember, this is launching from our custom AMI.

      So it won't just have the base Amazon Linux 2 operating system.

      Now it's going to have that base operating system plus all of the custom configuration that we did before creating the AMI.

      So rather than having to perform that same WordPress download installation configuration and the banner configuration each and every time, now we've baked that in to the AMI.

      So now when we launch one instance, 10 instances, or 100 instances from this AMI, all of them are going to have this configuration baked in.

      So let's give this a few minutes to launch.

      Once it's launched, we'll select it, right click, select Connect, and then connect into it using EC2, Instance Connect.

      Now one thing you will need to change because we're using a custom AMI, AWS can't necessarily detect the correct username to use.

      And so you might see sometimes it says root.

      Just go ahead and change this to EC2-user and then go ahead and click Connect.

      And if everything goes well, you'll be connected into the instance and you'll see our custom Cowsay banner.

      So all that configuration is now baked in and it's automatically included whenever we use that AMI to launch an instance.

      If we go back to the AWS console and select instances, make sure we still have the instance from AMI selected and then locate its public IP version for address.

      Don't use this link because that will use HTTPS instead, copy the IP address into your clipboard and open that in a new tab.

      Again, all being well, you should see the WordPress installation dialogue and that's because we've baked in the installation and the configuration into this AMI.

      So we've massively reduced the ongoing efforts required to launch an animals for life standard build configuration.

      If we use this AMI to launch hundreds or thousands of instances each and every time we're saving all the time and the effort required to perform this configuration and using an AMI is just one way that we can automate the build process of EC2 instances within AWS.

      And over the remainder of the course, I'm going to be demonstrating the other ways that you can use as well as comparing and contrasting the advantages and disadvantages of each of those methods.

      Now that's everything that I wanted to cover in this demo lesson.

      You've learned how to create an AMI and how to use it to save significant effort on an ongoing basis.

      So let's clear up all of the infrastructure that we've used in this lesson.

      So move back to the AWS console, close down this tab, go back to instances, and we need to manually terminate the instance that we created from our custom AMI.

      So right click and then go to terminate instance.

      You'll need to confirm that.

      That will start the process of termination.

      Now we're not going to delete the AMI or snapshots because there's a demo coming up later in this section of the course where you're going to get the experience of copying and sharing an AMI between AWS regions.

      So we're going to need to leave this in place.

      So we're not going to delete the AMI or the snapshots created within this lesson.

      Verify that that instance has been terminated and once it has, click on services, go to cloud formation, select the AMI demo stack, select delete and then confirm that deletion.

      And that will remove all of the infrastructure that we've created within this demo lesson.

      And at this point, that's everything that I wanted you to do in this demo.

      So go ahead, complete this video.

      And when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back and in this demo lesson you'll be creating an AMI from a pre-configured EC2 instance.

      So you'll be provisioning an EC2 instance, configuring it with a popular web application stack and then creating an AMI of that pre-configured web application.

      Now you know in the previous demo where I said that you would be implementing the WordPress manual install once?

      Well I might have misled you slightly but this will be the last manual install of WordPress in the course, I promise.

      What we're going to do together in this demo lesson is create an Amazon Linux AMI for the animals for life business but one which includes some custom configuration and an install of WordPress ready and waiting to be initially configured.

      So this is a fairly common use case so let's jump in and get started.

      Now in order to perform this demo you're going to need some infrastructure, make sure you're logged into the general AWS account, so the management account of the organization and as always make sure that you have the Northern Virginia region selected.

      Now attached to this lesson is a one-click deployment link, go ahead and click that link.

      This will open the quick create stack screen, it should automatically be populated with the AMI demo as the stack name, just scroll down to the bottom, check this capabilities acknowledgement box and then click on create stack.

      We're going to need this stack to be in a create complete state so go ahead and pause the video and we can resume once the stack moves into create complete.

      Okay so that stacks now moved into a create complete state, we're good to continue with the demo.

      Now you're going to be using some command line commands within an EC2 instance as part of creating an Amazon machine image so also attached to this lesson is the lessons command document which contains all of those commands so go ahead and open that document.

      Now you might recognize these as the same commands that you used when you were performing a manual WordPress installation and that's the case we're running the same manual installation process as part of setting up our animals for life AMI so you're going to need all of these commands but as you've already experienced them in the previous demo lesson I'm going to run through them a lot quicker in this demo lesson so go back to the AWS console and we need to move to the EC2 area of the console so click on the services drop down, type EC2 into this search box and then open that in a new tab.

      Once you there go ahead and click on running instances, close down any dialogues about any console changes we want to maximize the amount of screen space that we have, we're going to connect to this A4L public EC2 instance this is the instance that we're going to use to create our AMI so we're going to set the instance up manually how we want it to be and then we're going to use it to generate an AMI so we need to connect to this instance so right click select connect we're going to use EC2 instance connect to do the work within our browser so make sure the username is EC2-user and then connect to this instance then once connected we're going to run through the commands to install WordPress really quickly we're going to start again by setting the variables that will use throughout the installation so you can just go ahead and copy and paste those straight in and press enter now we're going to run through all of the next set of commands really quickly because you use them in the previous demo lesson so first we're going to go ahead and install the MariaDB server Apache and the Wget utility while that's installing copy all of the commands from step 3 so these are commands which enable and start Apache and MariaDB go ahead and paste all of those four in and press enter so now Apache and MariaDB are both set to start when the instance boots as well as being set to currently started I'll just clear the screen to make this easier to see next we're going to set the DB root password again that's this command using the contents of the variable that you set at the start next we download WordPress once it's downloaded we move into the web root folder we extract the download we copy the files from within the WordPress folder that we've just extracted into the current folder which is the web root once we've done that we remove the WordPress folder itself and then we tidy up by deleting the download I'm going to clear the screen we copy the template configuration file into its final file name so wp-config.php then we're going to replace the placeholders in that file we're going to start with the database name using the variable that you set at the start next we're going to use the database user which you also set at the start and finally the database password and then we're going to set the ownership on all of these files to be the Apache user and the Apache group clear the screen next we need to create the DB setup script that are demonstrated in the previous demo so we need to run a collection of commands the first to enter the create database command the next one to enter the create user command and set that password the next one to grant permissions on the database to that user then flush the permissions then we need to run that script using the MySQL command line interface that runs all of those commands and performs all of those operations and then we tidy up by deleting that file now at this point we've done the exact same process that we did in the previous demo we've installed and set up WordPress and if everything's working okay we can go back to the AWS console click on instances select the running a4l-public ec2 instance copy down its IP address again make sure you copy that down don't click this link and then open that in a new tab if everything's working as expected you should see the WordPress installation dialogue now this time because we're creating an AMI we don't want to perform the installation we want to make sure that when anyone uses this AMI they're also greeted with this installation so we're going to leave this at this point we're not going to perform the installation instead we're going to go back to the ec2 instance now because this ec2 instance is for the animals for life business we want to customize it and make sure that everybody knows that this is an animals for life ec2 instance now to do that we're going to install an animal themed utility called cow say I'm going to clear the screen to make it easier to see and then just to demonstrate exactly what cow say does I'm going to run a cow say oh hi and if all goes well we see a cow using ASCII art saying the oh hi message that we just typed so we're going to use this to create a message of the day welcome when anyone connects to this ec2 instance to do that we're going to create a file inside the configuration folder of this ec2 instance so we're going to use shudu nano and we're going to create this file so forward slash etc forward slash update hyphen motd dot d forward slash 40 hyphen cow so we're going to create that file this is the file that's going to be used to generate the output when anyone logs in to this ec2 instance so we're going to copy in these two lines and then press enter so this means when anyone logs into the ec2 instance they're going to get an animal themed welcome so use control o to save that file and control x to exit clear the screen to make it easier to see we're going to make sure that file that we've just edited has the correct permissions then we're going to force an update of the message of the day so this is going to be what's displayed when anyone logs into this instance and then finally now that we've completed this configuration we're going to reboot this ec2 instance so we're going to use this command to reboot it and just to illustrate how this works I'm going to close down that tab and return to the ec2 console give this a few moments to restart that should have rebooted by now so we're going to select it right click go to connect again use ec2 instance connect assuming everything's working now when we connect to the instance we'll see an animal themed login banner so this is just a nice way that we can ensure that anyone logging into this instance understands that a he uses the Amazon Linux 2 AMI and be that it belongs to animals for life so we've created this instance using the Amazon Linux 2 AMI we've performed the WordPress installation and initial configuration we've customized the banner and now we're going to use this as our template instance to create our AMI that can then be used to launch other instances okay so this is the end of part one of this lesson it was getting a little bit on the long side and so I wanted to add a break it's an opportunity just to take a rest or grab a coffee part 2 will be continuing immediately from the end of part one so go ahead complete the video and when you're ready join me in part two

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      So this is the folder containing the WordPress installation files.

      Now there's one particular file that's really important, and that's the configuration file.

      So there's a file called WP-config-sample, and this is actually the file that contains a template of the configuration items for WordPress.

      So what we need to do is to take this template and change the file name to be the proper file name, so wp-config.php.

      So we're going to create a copy of this file with the correct name.

      And to do that, we run this command.

      So we're copying the template or the sample file to its real file name, so wp-config.php.

      And this is the name that WordPress expects when it initially loads its configuration information.

      So run that command, and that now means that we have a live config file.

      Now this command isn't in the instructions, but if I just take a moment to open up this file, you don't need to do this.

      I'm just demonstrating what's in this file for your benefit.

      But if I run a sudo nano, and then wp, and then hyphen-config, and then php, this is how the file looks.

      So this has got all the configuration information in.

      So it stores the database name, the database user, the database host, and lots of other information.

      Now notice how it has some placeholders.

      So this is where we would need to replace the placeholders with the actual configuration information.

      So the database name itself, the host name, the database username, the database password, all that information would need to be replaced.

      Now we're not going to type this in manually, so I'm going to control X to exit out of this, and then clear the screen again to make it easy to see.

      We're going to use the Linux utility sed, or S-E-D.

      And this is a utility which can perform a search and replace within a text file.

      It's actually much more complex and capable than that.

      It can perform many different manipulation operations.

      But for this demonstration, we're going to use it as a simple search and replace.

      Now we're going to do this a number of times.

      First, we're going to run this command, which is going to replace this placeholder.

      Remember, this is one of the placeholders inside the configuration file that I've just demonstrated, wp-config.

      We're going to replace the placeholder here with the contents of the variable name, dbname, that we set at the start of this demo.

      So this is going to replace the placeholder with our actual database name.

      So I'm going to enter that so you can do the same.

      We're going to run the sed command again, but this time it's going to replace the username placeholder with the dbuser variable that we set at the start of this demo.

      So use that command as well.

      And then lastly, it will do the same for the database password.

      So type or copy and paste this command and press enter.

      And that now means that this wp-config has the actual configuration information inside.

      And just to demonstrate that, you don't need to do this part.

      I'll just do it to demonstrate.

      If I edit this file again, you'll see that all of these placeholders have actually been replaced with actual values.

      So I'm going to control X out of that and then clear the screen.

      And that concludes the configuration for the WordPress application.

      So now it's ready.

      Now it knows how to communicate with the database.

      What we need to do to finish off the configuration though is just to make sure that the web server has access to all of the files within this folder.

      And to do that, we use this command.

      So we're making sure that we use the shown command or chown and set the ownership of all of the files in this folder and any subfolders to be the Apache user and the Apache group.

      And the Apache user and Apache group belong to the web server.

      So this just makes sure that the web server is able to access and control all of the files in the web root folder.

      So run that command and press enter.

      And that concludes the installation part of the WordPress application.

      There's one final thing that we need to do and that's to create the database that WordPress will use.

      So I'm going to clear the screen to make it easy to see.

      Now what we're going to do in order to configure the database is we're going to make a database setup script.

      We're going to put this script inside the forward slash TMP folder and we're going to call it DB.setup.

      So what we need to do is enter the commands into this file that will create the database.

      After the database is created, it needs to create a database user and then it needs to grant that user permissions on that database.

      Now again, instead of manually entering this, we're going to use those variable names that were created at the start of the demo.

      So we're going to run a number of commands.

      These are all in the lessons commands document.

      The first one is this.

      So this echoes this text and because it has a variable name in, this variable name will be replaced by the actual contents of the variable.

      Then it's going to take this text with the replacement of the contents of this variable and it's going to enter that into this file.

      So forward slash TMP, forward slash DB setup.

      So run that and that command is going to create the WordPress database.

      Then we're going to use this command and this is the same so it echoes this text but it replaces these variable names with the contents of the variables.

      This is going to create our WordPress database user.

      It's going to set its password and then it's going to append this text to the DB setup file that we're creating.

      Now all of these are actually database commands that we're going to execute within the MariaDB database.

      So enter that to add that line to DB.setup.

      Then we have another line which uses the same architecture as the ones above it.

      It echoes the text.

      It replaces these variable names with the contents and then outputs that to this DB.setup file and this command grants our database user permissions to our WordPress database.

      And then the last command is this one which just flushes the privileges and again we're going to add this to our DB.setup script.

      So now I'm just going to cat the contents of this file so you can just see exactly what it looks like.

      So cat and then space forward slash TMP, forward slash DB.setup.

      So as you'll see it's replaced all of these variable names with the actual contents.

      So this is what the contents of this script actually looks like.

      So these are commands which will be run by the MariaDB database platform.

      To run those commands we use this.

      So this is the MySQL command line interface.

      So we're using MySQL to connect to the MariaDB database server.

      We're using the username of root.

      We're passing in the password and then using the contents of the DB root password variable.

      And then once we authenticate the database we're passing in the contents of our DB.setup script.

      And so this means that all of the lines of our DB.setup script will be run by the MariaDB database and this will create the WordPress database, the WordPress user and configure all of the required permissions.

      So go ahead and press enter.

      That command is run by the MariaDB platform and that means that our WordPress database has been successfully configured.

      And then lastly just to keep things secure because we don't want to leave files laying around on the file system with authentication information inside.

      We're just going to run this command to delete this DB.setup file.

      Okay, so that concludes the setup process for WordPress.

      It's been a fairly long intensive process but that now means that we have an installation of WordPress on this EC2 instance, a database which has been installed and configured.

      So now what we can do is to go back to the AWS console, click on instances.

      We need to select the A4L-PublicEC2 and then we need to locate its IP address.

      Now make sure that you don't use this open address link because this will attempt to open the IP address using HTTPS and we don't have that configured on this WordPress instance.

      Instead, just copy the IP address into your clipboard and then open that in a new tab.

      If everything's successful, you should see the WordPress installation dialog and just to verify this is working successfully, let's follow this process through.

      So pick English, United States for the language.

      For the blog title, just put all the cats and then admin as the username.

      You can accept the default strong password.

      Just copy that into your clipboard so we can use it to log in in a second and then just go ahead and enter your email.

      It doesn't have to be a correct one.

      So I normally use test@test.com and then go ahead and click on install WordPress.

      You should see a success dialog.

      Go ahead and click on login.

      Username will be admin, the password that you just copied into your clipboard and then click on login.

      And there you go.

      We've got a working WordPress installation.

      We're not going to configure it in any detail but if you want to just check out that it works properly, go ahead and click on this all the cats at the top and then visit site and you'll be able to see a generic WordPress blog.

      And that means you've completed the installation of the WordPress application and the database using a monolithic architecture on a single EC2 instance.

      So this has been a slow process.

      It's been manual and it's a process which is wide open for mistakes to be made at every point throughout that process.

      Can you imagine doing this twice?

      What about 10 times?

      What about a hundred times?

      It gets pretty annoying pretty quickly.

      In reality, this is never done manually.

      We use automation or infrastructure as code systems such as cloud formation.

      And as we move through the course, you're going to get experience of using all of these different methods.

      Now that we're close to finishing up the basics of VPC and EC2 within the course, things will start to get much more efficient quickly because I'm going to start showing you how to use many of the automation and infrastructure as code services within AWS.

      And these are really awesome to use.

      And you'll see just how much power is granted to an architect, a developer, or an engineer by using these services.

      For now though, that is the end of this demo lesson.

      Now what we're going to do is to clear up our account.

      So we need to go ahead and clear all of this infrastructure that we've used throughout this demo lesson.

      To do that, just move back to the AWS console.

      If you still have the cloud formation tab open and move back to that tab, otherwise click on services and then click on cloud formation.

      If you don't see it anywhere, you can use this box to search for it, select the word, press stack, select delete, and then confirm that deletion.

      And that will delete the stack, clear up all of the infrastructure that we've used throughout this demo lesson and the account will now be in the same state as it was at the start of this lesson.

      So from this point onward in the course, we're going to start using automation.

      Now there is a lesson coming up in a little while in this section of the course, where you're going to create an Amazon machine image which is going to contain a pre-baked copy of the WordPress application.

      So as part of that lesson, you are going to be required to perform one more manual installation of WordPress, but that's going to be part of automating the installation.

      So you'll start to get some experience of how to actually perform automated installations and how to design architectures which have WordPress as a component.

      At this point though, that's everything I wanted to cover.

      So go ahead, complete this video, and when you're ready, I look forward to you joining me in the next.

    1. Welcome back and in this lesson we're going to be doing something which I really hate doing and that's using WordPress in a course as an example.

      Joking aside though WordPress is used in a lot of courses as a very simple example of an application stack.

      The problem is that most courses don't take this any further.

      But in this course I want to use it as one example of how an application stack can be evolved to take advantage of AWS products and services.

      What we're going to be using WordPress for in this demo is to give you experience of how a manual installation of a typical application stack works in EC2.

      We're going to be doing this so you can get the experience of how not to do things.

      My personal belief is that to fully understand the advantages that automation features within AWS provide, you need to understand what a manual installation is like and what problems you can experience doing that manual installation.

      As we move through the course we can compare this to various different automated ways of installing software within AWS.

      So you're going to get the experience of bad practices, good practices and the experience to be able to compare and contrast between the two.

      By the end of this demonstration you're going to have a working WordPress site but it won't have any high availability because it's running on a single EC2 instance.

      It's going to be architecturally monolithic with everything running on the one single instance.

      In this case that means both the application and the database.

      The design is fairly straightforward.

      It's just the Animals for Life VPC.

      We're going to be deploying the WordPress application into a single subnet, the WebA public subnet.

      So this subnet is going to have a single EC2 instance deployed into it and then you're going to be doing a manual install onto this instance and the end result is a working WordPress installation.

      At this point it's time to get started and implement this architecture.

      So let's go ahead and switch over to our AWS console.

      To get started with this demo lesson you're going to need to do a few preparation steps.

      First just make sure that you're logged in to the general AWS account, so the management account of the organization and as always make sure you have the Northern Virginia region selected.

      Now attached to this lesson is a one-click deployment for the base infrastructure that we're going to use.

      So go ahead and open the one-click deployment link that's attached to this lesson.

      That link is going to take you to the Quick Create Stack screen.

      Everything should be pre-populated.

      The stack name should be WordPress.

      All you need to do is scroll down towards the bottom, check this capabilities box and then click on Create Stack.

      And this stack is going to need to be in a Create Complete state before we move on with the demo lesson.

      So go ahead and pause this video, wait for the stack to change to Create Complete and then we're good to continue.

      Also attached to this lesson is a Lessons Command document which lists all of the commands that you'll be using within the EC2 instance throughout this demo lesson.

      So go ahead and open that as well.

      So that should look something like this and these are all of the commands that we're going to be using.

      So these are the commands that perform a manual WordPress installation.

      Now that that stack's completed and we've got the Lesson Commands document open, the next step is to move across to the EC2 console because we're going to actually install WordPress manually.

      So click on the Services drop-down and then locate EC2 in this All Services part of the screen.

      If you've recently visited it, it should be in the Recently Visited section under Favorites or you can go ahead and type EC2 in the search box and then open that in a new tab.

      And then click on Instances running and you should see one single instance which is called A4L-PublicEC2.

      Go ahead and right-click on this instance.

      This is the instance we'll be installing WordPress within.

      So right-click, select Connect.

      We're going to be using our browser to connect to this instance so we'll be using Instance Connect just verify that the username is EC2-user and then go ahead and connect to this instance.

      Now again, I fully understand that a manual installation of WordPress might seem like a waste of time but I genuinely believe that you need to understand all the problems that come from manually installing software in order to understand the benefits which automation provides.

      It's not just about saving time and effort.

      It's also about error reduction and the ability to keep things consistent.

      Now I always like to start my installations or my scripts by setting variables which will store the configuration values that everything from that point forward will use.

      So we're going to create four variables.

      One for the database name, one for the database user, one for the database password and then one for the root or admin password of the database server.

      So let's start off by using the pre-populated values from the Lessened Commands documents.

      So that's all of those variables set and we can confirm that those are working by typing echo and then a space and then a dollar and then the name of one of those variables.

      So for example, dbname and press Enter and that will show us the value stored within that variable.

      So now we can use these later points of the installation.

      So at this point I'm going to clear the screen to keep it easy to see and stage two at this installation process is to install some system software.

      So there are a few things that we need to install in order to allow a WordPress installation.

      We'll install those using the DNF package manager.

      We need to give it admin privileges which is why we use shudu and then the packages that we're going to install are the database server which is Maria db-server the Apache web server which is HTTPD and then a utility called Wget which we're going to use to download further components of the installation.

      So go ahead and type or copy and paste that command and press Enter and that installation process will take a few moments and it will go through installing that software and any of the prerequisites.

      They're done so I'll clear the screen to keep this easy to read.

      Now that all those packages are installed we need to start both the web server and the database server and ensure that both of them are started if ever the machine is restarted.

      So to do that we need to enable and start those services.

      So enabling and starting means that both of the services are both started right now and they'll start if the machine reboots.

      So first we'll use this command.

      So we're using admin privileges again, systemctl which allows us to start and stop system processes and then we use enable and then HTTPD which is the web server.

      So type and press enter and that ensures that the web server is enabled.

      We need to run the same command again but this time specifying MariaDB to ensure that the database server is enabled.

      So type or copy and paste and press enter.

      So that means both of those processes will start if ever the instance is rebooted and now we need to manually start both of those so they're running and we can interact with them.

      So we need to use the same structure of command but instead of enable we need to start both of these processes.

      So first the web server and then the database server.

      So that means the CC2 instance now has a running web and database server both of which are required for WordPress.

      So I'll clear the screen to keep this easy to read.

      Next we're going to move to stage 4 and stage 4 is that we need to set the root password of the database server.

      So this is the username and password that will be used to perform all of the initial configuration of the database server.

      Now we're going to use this command and you'll note that for password we're actually specifying one of the variables that we configured at the start of this demo.

      So we're using the DB root password variable that we configured right at the start.

      So go ahead and copy and paste or type that in and press enter and that sets the password for the root user of this database platform.

      The next step which is step 5 is to install the WordPress application files.

      Now to do that we need to install these files inside what's known as the web root.

      So whenever you browse to a web server either using an IP address or a DNS name if you don't specify a path so if you just use the server name for example netflix.com then it loads those initial files from a folder known as the web root.

      Now on this particular server the web root is stored in /varr/www/html so we need to download WordPress into that folder.

      Now we're going to use this command Wget and that's one of the packages that we installed at the start of this lesson.

      So we're giving it admin privileges and we're using Wget to download latest.tar.gz from wordpress.org and then we're putting it inside this web root.

      So /varr/www/html.

      So go ahead and copy and paste or type that in and press enter.

      That'll take a few moments depending on the speed of the WordPress servers and that will store latest.tar.gz in that web root folder.

      Next we need to move into that folder so cd space /varr/www/html and press enter.

      We need to use a Linux utility called tar to extract that file.

      So sudo and then tar and then the command line options -zxvf and then the name of the file so latest.tar.gz So copy and paste or type that in and press enter and that will extract the WordPress download into this folder.

      So now if we do an ls -la you'll see that we have a WordPress folder and inside that folder are all of the application files.

      Now we actually don't want them inside a WordPress folder.

      We want them directly inside the web root.

      So the next thing we're going to do is this command and this is going to copy all of the files from inside this WordPress folder to . and . represents the current folder.

      So it's going to copy everything inside WordPress into the current working directory which is the web root directory.

      So enter that and that copies all of those files.

      And now if we do another listing you'll see that we have all of the WordPress application files inside the web root.

      And then lastly for the installation part we need to tidy up the mess that we've made.

      So we need to delete this WordPress folder and the download file that we just created.

      So to do that we'll run an rm -r and then WordPress to delete that folder.

      And then we'll delete the download with sudo rm and then a space and then the name of the file.

      So latest.tar.gz.

      And that means that we have a nice clean folder.

      So I'll clear the screen to make it easy to see.

      And then I'll just do another listing.

      Okay so this is the end of part one of this lesson.

      It was getting a little bit on the long side and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part two will be continuing immediately from the end of part one.

      So go ahead complete the video and when you're ready join me in part two.

    1. Welcome back and in this video we're going to interact with instant store volumes.

      Now this part of the demo does come at a cost.

      This isn't inside the free tier because we're going to be launching some instances which are fairly large and are not included in the free tier.

      The demo has a cost of approximately 13 cents per hour and so you should only do this part of the demo if you're willing to accept that cost.

      If you don't want to accept those costs then you can go ahead and watch me perform these within my test environment.

      So to do this we're going to go ahead and click on instances and we're going to launch an instance manually.

      So I'm going to click on launch instances.

      We're going to name the instance, Instance Store Test so put that in the name box.

      Then scroll down, pick Amazon Linux, make sure Amazon Linux 2023 is selected and the architecture needs to be 64 bit x86.

      Scroll down and then in the instance type box click and we need to find a different type of instance.

      This is going to be one that supports instance store volumes.

      So scroll down and we're looking for m5dn.large.

      This is a type of instance which includes one instance store volume.

      So select that then scroll down a little bit more and under key pair click in the box and select proceed without a key pair not recommended.

      Scroll down again and under network settings click on edit.

      Click in the VPC drop down and select a4l-vpc1.

      Under subnet make sure sn-web-a is selected.

      Make sure enabled is selected for both of the auto assign public IP drop downs.

      Then we're going to select an existing security group click the drop down select the EBS demo instance security group.

      It will have some random after it but that's okay.

      Then scroll down and under storage we're going to leave all of the defaults.

      What you are able to do though is to click on show details next to instance store volumes.

      This will show you the instance store volumes which are included with this instance.

      You can see that we have one instance store volume it's 75 GB in size and it has a slightly different device name.

      So dev nvme0n1.

      Now all of that looks good so we're just going to go ahead and click on launch instance.

      Then click on view all instances and initially it will be an appending state and eventually it will move into a running state.

      Then we should probably wait for the status check column to change from initializing to 2 out of 2.

      Go ahead and pause the video and wait for this status check to change to be fully green.

      It should show 2 out of 2 status checks.

      That's now in a running state with 2 out of 2 checks so we can go ahead and connect to this instance.

      Before we do though just go ahead and select the instance and just note the instances public IP version 4 address.

      Now this address is really useful because it will change if the EC2 instance moves between EC2 hosts.

      So it's a really easy way that we can verify whether this instance has moved between EC2 hosts.

      So just go ahead and note down the IP address of the instance that you have if you're performing this in your own environment.

      We're going to go ahead and connect to this instance though so right click, select connect, we'll be choosing instance connect, go ahead and connect to the instance.

      Now many of these commands that we'll be using should by now be familiar.

      Just refer back to the lessons command document if you're unsure because we'll be using all of the same commands.

      First we need to list all of the block devices which are attached to this instance and we can do that with LSBLK.

      This time it looks a little bit different because we're using instance store rather than EBS additional volumes.

      So in this particular case I want you to look for the 8G volume so this is the root volume.

      This represents the boot or root volume of the instance.

      Remember that this particular instance type came with a 75GB instance store volume so we can easily identify it's this one.

      Now to check that we can verify whether there's a file system on this instance store volume.

      If we run this command, so the same command we've used previously so shudu file -s and then the id of this volume so dev nvme1n1, you'll see it reports data.

      And if you recall from the previous parts of this demo series this indicates that there isn't a file system on this volume.

      We're going to create one and to do that we use this command again it's the same command that we've used previously just with the new volume id.

      So press enter to create a file system on this raw block device this instance store volume and then we can run this command again to verify that it now has a file system.

      To mount it we can follow the same process that we did in the earlier stages of this demo series.

      We'll need to create a directory for this volume to be mounted into this time we'll call it forward slash instance store.

      So create that folder and then we're going to mount this device into that folder so shudu mount then the device id and then the mount point or the folder that we've previously created.

      So press enter and that means that this block device this instance store volume is now mounted into this folder.

      And if we run a df space -k and press enter you can see that it's now mounted.

      Now we're going to move into that folder by typing cd space forward slash instance store and to keep things efficient we're going to create a file called instance store dot txt.

      And rather than using an editor we'll just use shudu touch and then the name of the file and this will create an empty file.

      If we do an LS space -la and press enter you can see that that file exists.

      So now that we have this file stored on a file system which is running on this instance store volume let's go ahead and reboot this instance.

      Now we need to be careful we're not going to stop and start the instance we're going to restart the instance.

      Restarting is different than stop and start.

      So to do that we're going to close this tab move back to the ec2 console so click on instances right click on instance store test and select reboot instance and then confirm that.

      Note what this IP address is before you initiate the reboot operation and then just give this a few minutes to reboot.

      Then right click and select connect.

      Using instance connect go ahead and connect back to the instance.

      And again if it appears to hang at this point then you can just wait for a few moments and then connect again.

      But in this case I've left it long enough and I'm connected back into the instance.

      Now once I'm back in the instance if I run a df space -k and press enter note how that file system is not mounted after the reboot.

      Now that's fine because we didn't configure the Linux operating system to mount this file system when the instance is restarted.

      But what we can do is do an LS BLK again to list the block device.

      We can see that it's still there and we can manually mount it back in the same folder as it was before the reboot.

      To do that we run this command.

      So it's mounting the same volume ID the same device ID into the same folder.

      So go ahead and run that command and press enter.

      Then if we use cd space forward slash and then instance store press enter and then do an LS space -la we can see that this file is still there.

      Now the file is still there because instance store volumes do persist through the restart of an EC2 instance.

      Restarting an EC2 instance does not move the instance from one EC2 host to another.

      And because instance store volumes are directly attached to an EC2 host this means that the volume is still there after the machine has restarted.

      Now we're going to do something different though.

      Close this tab down.

      Move back to instances.

      Again pay special attention to this IP address.

      Now we're going to right click and stop the instance.

      So go ahead and do that and confirm it if you're doing this in your own environment.

      Watch this public IP v4 address really carefully.

      We'll need to wait for the instance to move into a stopped state which it has and if we select the instance note how the public IP version for address has been unallocated.

      So this instance is now not running on an EC2 host.

      Let's right click.

      Go to start instance and start it up again.

      Only to give that a few moments again.

      It'll move into a running state but notice how the public IP version for address has changed.

      This is a good indication that the instance has moved from one EC2 host to another.

      So let's give this instance a few moments to start up.

      And once it has right click, select connect and then go ahead and connect to the instance using instance connect.

      Once connected go ahead and run an LS BLK and press enter and you'll see it appears to have the same instance store volume attached to this instance.

      It's using the same ID and it's the same size.

      But let's go ahead and verify the contents of this device using this command.

      So shudu file space -s space and then the device ID of the instance store volume.

      For press enter, now note how it shows data.

      So even though we created a file system in the previous step after we've stopped and started the instance, it appears this instance store volume has no data.

      Now the reason for that is when you restart an EC2 instance, it restarts on the same EC2 host.

      But when you stop and start an EC2 instance, which is a distinctly different operation, the EC2 instance moves from one EC2 host to another.

      And that means that it has access to completely different instance store volumes than it did on that previous host.

      It means that all of the data, so the file system and the test file that we created on the instance store volume, before we stopped and started this instance, all of that is lost.

      When you stop and start an EC2 instance or for any other reason, which means the instance moves from one host to another, all of the data is lost.

      So instance store volumes are ephemeral.

      They're not persistent and you can't rely on them to keep your data safe.

      And it's really important that you understand that distinction.

      If you're doing the developer or sysop streams, it's also important that you understand the difference between an instance restart, which keeps the same EC2 host, and a stop and start, which moves an instance from one host to another.

      The format means you're likely to keep your data, but the latter means you're guaranteed to lose your data when using instance store volumes.

      EBS on the other hand, as we've seen, is persistent and any data persists through the lifecycle of an EC2 instance.

      Now with that being said, though, that's everything that I wanted to demonstrate within this series of demo lessons.

      So let's go ahead and tidy up the infrastructure.

      Close down this tab, click on instances.

      If you follow this last part of the demo in your own environment, go ahead and right click on the instance store test instance and terminate that instance.

      That will delete it along with any associated resources.

      We'll need to wait for this instance to move into the terminated state.

      So give that a few moments.

      Once that's terminated, go ahead and click on services and then move back to the cloud formation console.

      You'll see the stack that you created using the one click deploy at the start of this lesson.

      Go ahead and select that stack, click on delete and then delete stack.

      And that's going to put the account back in the same state as it was at the start of this lesson.

      So it will remove all of the resources that have been created.

      And at that point, that's the end of this demo series.

      So what did you learn?

      You learned that EBS volumes are created within one specific availability zone.

      EBS volumes can be mounted to instances in that availability zone only and can be moved between instances while retaining their data.

      You can create a snapshot from an EBS volume which is stored in S3 and that data is replicated within the region.

      And then you can use snapshots to create volumes in different availability zones.

      I told you how snapshots can be copied to other AWS regions either as part of data migration or disaster recovery and you learned that EBS is persistent.

      You've also seen in this part of the demo series that instant store volumes can be used.

      They are included with many instance types but if the instance moves between EC2 hosts so if an instance is stopped and then started or if an EC2 host has hardware problems then that EC2 instance will be moved between hosts and any data on any instant store volumes will be lost.

      So that's everything that you needed to know in this demo lesson and you're going to learn much more about EC2 and EBS in other lessons throughout the course.

      At this point though, thanks for watching and doing this demo.

      I hope it was useful but go ahead complete this video and when you're ready I look forward to you joining me in the next.

    1. Welcome back and we're going to use this demo lesson to get some experience of working with EBS and Instant Store volumes.

      Now before we get started, this series of demo videos will be split into two main components.

      The first component will be based around EBS and EBS snapshots and all of this will come under the free tier.

      The second component will be based on Instant Store volumes and will be using larger instances which are not included within the free tier.

      So I'm going to make you aware of when we move on to a part which could incur some costs and you can either do that within your own environment or watch me do it in the video.

      If you do decide to do it in your own environment, just be aware that there will be a 13 cents per hour cost for the second component of this demo series and I'll make it very clear when we move into that component.

      The second component is entirely optional but I just wanted to warn you of the potential cost in advance.

      Now to get started with this demo, you're going to need to deploy some infrastructure.

      To do that, make sure that you're logged in to the general account, so the management account of the organization and you've got the Northern Virginia region selected.

      Now attached to this demo is a one click deployment link to deploy the infrastructure.

      So go ahead and click on that link.

      That's going to open this quick create stack screen and all you need to do is scroll down to the bottom, check this capabilities box and click on create stack.

      Now you're going to need this to be in a create complete state before you continue with this demo.

      So go ahead and pause the video, wait for that stack to move into the create complete status and then you can continue.

      Okay, now that's finished and the stack is in a create complete state.

      You're also going to be running some commands within EC2 instances as part of this demo.

      Also attached to this lesson is a lesson commands document which contains all of those commands and you can use this to copy and paste which will avoid errors.

      So go ahead and open that link in a separate browser window or separate browser tab.

      It should look something like this and we're going to be using this throughout the lesson.

      Now this cloud formation template has created a number of resources, but the three that we're concerned about are the three EC2 instances.

      So instance one, instance two and instance three.

      So the next thing to do is to move across to the EC2 console.

      So click on the services drop down and then either locate EC2 under all services, find it in recently visited services or you can use the search box at the top type EC2 and then open that in a new tab.

      Now the EC2 console is going through a number of changes so don't be alarmed if it looks slightly different or if you see any banners welcoming you to this new version.

      Now if you click on instances running, you'll see a list of the three instances that we're going to be using within this demo lesson.

      We have instance one - az a.

      We have instance two - az a and then instance one - az b.

      So these are three instances, two of which are in availability zone A and one which is in availability zone B.

      Next I want you to scroll down and locate volumes under elastic block store and just click on volumes.

      And what you'll see is three EBS volumes, each of which is eight GIB in size.

      Now these are all currently in use.

      You can see that in the state column and that's because all of these volumes are in use as the boot volumes for those three EC2 instances.

      So on each of these volumes is the operating system running on those EC2 instances.

      Now to give you some experience of working with EBS volumes, we're going to go ahead and create a volume.

      So click on the create volume button.

      The first thing you'll need to do when creating a volume is pick the type and there are a number of different types available.

      We've got GP2 and GP3 which are the general purpose storage types.

      We're going to use GP3 for this demo lesson.

      You could also select one of the provisioned IOPS volumes.

      So this is currently IO1 or IO2.

      And with both of these volume types, you're able to define IOPS separately from the size of the volume.

      So these are volume types that you can use for demanding storage scenarios where you need high-end performance or when you need especially high performance for smaller volume sizes.

      Now IO1 was the first type of provisioned IOPS SSD introduced by AWS and more recently they've introduced IO2 and enhanced it which provides even higher levels of performance.

      In addition to that we do have the non-SSD volume types.

      So SC1 which is cold HDD, ST1 which is throughput optimized HDD and then of course the original magnetic type which is now legacy and AWS don't recommend this for any production usage.

      For this demo lesson we're going to go ahead and select GP3.

      So select that.

      Next you're able to pick a size in GIB for the volume.

      We're going to select a volume size of 10 GIB.

      Now EBS volumes are created within a specific availability zone so you have to select the availability zone when you're creating the volume.

      At this point I want you to go ahead and select US-EAST-1A.

      When creating volume you're also able to specify a snapshot as the basis for that volume.

      So if you want to restore a snapshot into this volume you can select that here.

      At this stage in the demo we're going to be creating a blank EBS volume so we're not going to select anything in this box.

      We're going to be talking about encryption later in this section of the course.

      You are able to specify encryption settings for the volume when you create it but at this point we're not going to encrypt this volume.

      We do want to add a tag so that we can easily identify the volume from all of the others so click on add tag.

      For the key we're going to use name.

      For the value we're going to put EBS test volume.

      So once you've entered both of those go ahead and click on create volume and that will begin the process of creating the volume.

      Just close down any dialogues and then just pay attention to the different states that this volume goes through.

      It begins in a creating state.

      This is where the storage is being provisioned and then made available by the EBS product.

      If we click on refresh you'll see that it changes from creating to available and once it's in an available state this means that we can attach it to EC2 instances.

      And that's what we're going to do so we're going to right click and select attach volume.

      Now you're able to attach this volume to EC2 instances but crucially only those in the same availability zone.

      EBS is an availability zone scoped service and so you can only attach EBS volumes to EC2 instances within that same availability zone.

      So if we select the instance box you'll only see instances in that same availability zone.

      Now at this point go ahead and select instance 1 in availability zone A.

      Once you've selected it you'll see that the device field is populated and this is the device ID that the instance will see for this volume.

      So this is how the volume is going to be exposed to the EC2 instance.

      So if we want to interact with this instance inside the operating system this is the device that we'll use.

      Now different operating systems might see this in slightly different ways.

      So as this warning suggests certain Linux kernels might rename SDF to XVDF.

      So we've got to be aware that when you do attach a volume to an EC2 instance you need to get used to how that's seen inside the operating system.

      How we can identify it and how we can configure it within the operating system for use.

      And I'm going to demonstrate that in the next part of this demo lesson.

      So at this point just go ahead and click on attach and this will attach this volume to the EC2 instance.

      Once that's attached to the instance and you see the state change to in use then just scroll up on the left hand side and select instances.

      We're going to go ahead and connect to instance 1 in availability zone A.

      This is the instance that we just attached that EBS volume to so we want to interact with this instance and see how we can see the EBS volume.

      So right click on this instance and select connect and then you could either connect with an SSH client or use instance connect and to keep things simple we're going to connect from our browser so select the EC2 instance connect option make sure the user's name is set to EC2-user and then click on connect.

      So now we connected to this EC2 instance and it's at this point that we'll start needing the commands that are listed inside the lesson commands document and again this is attached to this lesson.

      So first we need to list all the block devices which are connected to this instance and we're going to use the LSBLK command.

      Now if you're not comfortable with Linux don't worry just take this nice and slowly and understand at a high level all the commands that we're going to run.

      So the first one is LSBLK and this is list block devices.

      So if we run this we'll be able to see a list of all of the block devices connected to this EC2 instance.

      You'll see the root device this is the device that's used to boot the instance it contains the instance operating system you'll see that it's 8 gig in size and then this is the EBS volume that we just attached to this instance.

      You'll see that device ID so XVDF and you'll see that it's 10 gig in size.

      Now what we need to do next is check whether there is a file system on this block device.

      So this block device we've created it with EBS and then we've attached it to this instance.

      Now we know that it's blank but it's always safe to check if there's any file system on a block device.

      So to do that we run this command.

      So we're going to check are there any file systems on this block device.

      So press enter and if you see just data that indicates that there isn't any file system on this device and so we need to create one.

      You can only mount file systems under Linux and so we need to create a file system on this raw block device this EBS volume.

      So to do that we run this command.

      So shoo-doo again is just giving us admin permissions on this instance.

      MKFS is going to make a file system.

      We specify the file system type with hyphen t and then XFS which is a type of file system and then we're telling it to create this file system on this raw block device which is the EBS volume that we just attached.

      So press enter and that will create the file system on this EBS volume.

      We can confirm that by rerunning this previous command and this time instead of data it will tell us that there is now an XFS file system on this block device.

      So now we can see the difference.

      Initially it just told us that there was data, so raw data on this volume and now it's indicating that there is a file system, the file system that we just created.

      Now the way that Linux works is we mount a file system to a mount point which is a directory.

      So we're going to create a directory using this command.

      MKDIR makes a directory and we're going to call the directory forward slash EBS test.

      So this creates it at the top level of the file system.

      This signifies root which is the top level of the file system tree and we're going to make a folder inside here called EBS test.

      So go ahead and enter that command and press enter and that creates that folder and then what we can do is to mount the file system that we just created on this EBS volume into that folder.

      And to do that we use this command, mount.

      So mount takes a device ID, so forward slash dev forward slash xvdf.

      So this is the raw block device containing the file system we just created and it's going to mount it into this folder.

      So type that command and press enter and now we have our EBS volume with our file system mounted into this folder.

      And we can verify that by running a df space hyphen k.

      And this will show us all of the file systems on this instance and the bottom line here is the one that we've just created and mounted.

      At this point I'm just going to clear the screen to make it easier to see and what we're going to do is to move into this folder.

      So cd which is change directory space forward slash EBS test and then press enter and that will move you into that folder.

      Once we're in that folder we're going to create a test file.

      So we're going to use this command so shudu nano which is a text editor and we're going to call the file amazing test file dot txt.

      So type that command in and press enter and then go ahead and type a message.

      It can be anything you just need to recognize it as your own message.

      So I'm going to use cats are amazing and then some exclamation marks.

      Then I'm going to press control o and enter to save that file and then control x to exit again clear the screen to make it easier to see.

      And then I'm going to do an LS space hyphen LA and press enter just to list the contents of this folder.

      So as you can see we've now got this amazing test file dot txt.

      And if we cat the contents of this so cat amazing test file dot txt you'll see the unique message that you just typed in.

      So at this point we've created this file within the folder and remember the folder is now the mount point for the file system that we created on this EBS volume.

      So the next step that I want you to do is to reboot this EC2 instance.

      To do that type sudo space and then reboot and press enter.

      Now this will disconnect you from this session.

      So you can go ahead and close down this tab and go back to the EC2 console.

      Just go ahead and click on instances.

      Okay, so this is the end of part one of this lesson.

      It was getting a little bit on the long side and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part two will be continuing immediately from the end of part one.

      So go ahead complete the video and when you're ready join me in part two.

    1. Welcome back and in this demo lesson you're going to evolve the infrastructure which you've been using throughout this section of the course.

      In this demo lesson you're going to add private internet access capability using NAT gateways.

      So you're going to be applying a cloud formation template which creates this base infrastructure.

      It's going to be the animals for life VPC with infrastructure in each of three availability zones.

      So there's a database subnet, an application subnet and a web subnet in availability zone A, B and C.

      Now to this point what you've done is configured public subnet internet access and you've done that using an internet gateway together with routes on these public subnets.

      In this demo lesson you're going to add NAT gateways into each availability zone so A, B and C and this will allow this private EC2 instance to have access to the internet.

      Now you're going to be deploying NAT gateways into each availability zone so that each availability zone has its own isolated private subnet access to the internet.

      It means that if any of the availability zones fail then each of the others will continue operating because these route tables which are attached to the private subnets they point at the NAT gateway within that availability zone.

      So each availability zone A, B and C has its own corresponding NAT gateway which provides private internet access to all of the private subnets within that availability zone.

      Now in order to implement this infrastructure you're going to be applying a one-click deployment and that's going to create everything that you see on screen now apart from these NAT gateways and the route table configurations.

      So let's go ahead and move across to our AWS console and get started implementing this architecture.

      Okay so now we're at the AWS console as always just make sure that you're logged in to the general AWS account as the I am admin user and you'll need to have the Northern Virginia region selected.

      Now at the end of the previous demo lesson you should have deleted all of the infrastructure that you've created up until that point so the animals for live VPC as well as the Bastion host and the associated networking.

      So you should have a relatively clean AWS account.

      So what we're going to do first is use a one-click deployment to create the infrastructure that we'll need within this demo lesson.

      So attached to this demo lesson is a one-click deployment link so go ahead and open that link.

      That's going to take you to a quick create stack screen.

      Everything should be pre-populated the stack name should be a4l just scroll down to the bottom check this capabilities box and then click on create stack.

      Now this will start the creation process of this a4l stack and we will need this to be in a create complete state before we continue.

      So go ahead pause the video wait for your stack to change into create complete and then we good to continue.

      Okay so now this stacks moved into a create complete state then we good to continue.

      So what we need to do before we start is make sure that all of our infrastructure has finished provisioning.

      To do that just go ahead and click on the resources tab of this cloud formation stack and look for a4l internal test.

      This is an EC2 instance a private EC2 instance so this doesn't have any public internet connectivity and we're going to use this to test on that gateway functionality.

      So go ahead and click on this icon under physical ID and this is going to move you to the EC2 console and you'll be able to see this a4l - internal - test instance.

      Now currently in my case it's showing as running but the status check is showing as initializing.

      Now we'll need this instance to finish provisioning before we can continue with the demo.

      What should happen is this status check should change from initializing to two out of two status checks and once you're at that point you should be able to right click and select connect and choose session manager and then have the option of connecting.

      Now you'll see that I don't because this instance hasn't finished its provisioning process.

      So what I want you to do is to go ahead and pause this video wait for your status checks to change to two out of two checks and then just go ahead and try to connect to this instance using session manager.

      Only resume the video once you've been able to click on connect under the session manager tab and don't worry if this takes a few more minutes after the instance finishes provisioning before you can connect to session manager.

      So go ahead and pause the video and when you can connect to the instance you're good to continue.

      Okay so in my case it took about five minutes for this to change to two out of two checks past and then another five minutes before I could connect to this EC2 instance.

      So I can right click on here and put connect.

      I'll have the option now of picking session manager and then I can click on connect and this will connect me in to this private EC2 instance.

      Now the reason why you're able to connect to this private instance is because we're using session manager and I'll explain exactly how this product works elsewhere in the course but essentially it allows us to connect into an EC2 instance with no public internet connectivity and it's using VPC interface endpoints to do that which I'll be explaining elsewhere in the course but what you should find when you're connected to this instance if you try to ping any internet IP address so let's go ahead and type ping and then a space 1.1.1.1.1 and press enter you'll note that we don't have any public internet connectivity and that's because this instance doesn't have a public IP version for address and it's not in a subnet with a route table which points at the internet gateway.

      This EC2 instance has been deployed into the application a subnet which is a private subnet and it also doesn't have a public IP version for address.

      So at this point what we need to do is go ahead and deploy our NAT gateways and these NAT gateways are what will provide this private EC2 instance with connectivity to the public IP version for internet so let's go ahead and do that.

      Now to do that we need to be back at the main AWS console click in the services search box at the top type VPC and then right click and open that in a new tab.

      Once you do that go ahead and move to that tab once you there click on NAT gateways and create a NAT gateway.

      Okay so once you're here you'll need to specify a few things you'll need to give the NAT gateway a name you'll need to pick a public subnet for the NAT gateway to go into and then you'll need to give the NAT gateway an elastic IP address which is an IP address which doesn't change.

      So first we'll set the name of the NAT gateway and we'll choose to use a4l for animals for life -vpc1 -natgw and then -a because this is going into availability zone A.

      Next we'll need to pick the public subnet that the NAT gateway will be going into so click on the subnet drop down and then select the web a subnet which is the public subnet in availability zone a so sn -web -a.

      Now we need to give this NAT gateway an elastic IP it doesn't currently have one so we need to click on allocate elastic IP which gives it an allocation.

      Don't worry about the connectivity type we'll be covering that elsewhere in the course just scroll down to the bottom and create the NAT gateway.

      Now this process will take some time and so we need to go ahead and create the two other NAT gateways.

      So click on NAT gateways at the top and then we're going to create a second NAT gateway.

      So go ahead and click on create NAT gateway again this time we'll call the NAT gateway a4l -vpc1 -natgw -b and this time we'll pick the web b subnet so sn -web -b allocated elastic IP again and click on create NAT gateway then we'll follow the same process a third time so click create NAT gateway use the same naming scheme but with -c pick the web c subnet from the list allocate an elastic IP and then scroll down and click on create NAT gateway and at this point we've got the three NAT gateways that are being created they're all in appending state if we go to elastic IPs we can see the three elastic IPs which have been allocated to the NAT gateways and we can scroll to the right or left and see details on these IPs and if we wanted we could release these IPs back to the account once we'd finish with them now at this point you need to go ahead and pause the video and resume it once all three of those NAT gateways have moved away from appending state we need them to be in an available state ready to go before we can continue with this demo so go ahead and pause and resume once all three have changed to an available state okay so all these are now in an available state so that means they're good to go they're providing service now if you scroll to the right in this list you're able to see additional information about these NAT gateways so you can see the elastic and private IP address the VPC and then the subnet that each of these NAT gateways are located in what we need to do now is configure the routing so that the private instances can communicate via the NAT gateways so right click on route tables and open in a new tab and we need to create a new route table for each of the availability zones so go ahead and click on create route table first we need to pick the VPC for this route table so click on the VPC drop down and then select the animals for live VPC so a for L hyphen VPC one once selected go ahead and name at the route table we're going to keep the naming scheme consistent so a for L hyphen VPC one hyphen RT for route table hyphen private a so enter that and click on create then close that dialogue down and create another route table this time we'll use the same naming scheme but of course this time it will be RT hyphen private B select the animals for life VPC and click on create close that down and then finally click on create route table again this time a for L hyphen VPC one hyphen RT hyphen private C again click on the VPC drop down and select the animals for life VPC and then click on create so that's going to leave us with three route tables one for each availability zone what we need to do now is create a default route within each of these route tables and that route is going to point at the NAT gateway in the same availability zone so select the route table private a and then click on the routes tab once you've selected the routes tab click on edit routes and we're going to add a new route it's going to be the IP version for default route of 0.0.0.0/0 and then click on target and pick NAT gateway and we're going to pick the NAT gateway in availability zone a and because we named them it makes it easy to select the relevant one from this list so go ahead and pick a for L hyphen VPC one hyphen NAT GW hyphen a so because this is the route table in availability zone a we need to pick the same NAT gateway so save that and close and now we'll be doing the same process for the route table in availability zone B make sure the routes tab is selected and click on edit routes click on add route again 0.0.0.0/0 and then for target pick NAT gateway and then pick the NAT gateway that's in availability zone B so NAT GW hyphen B once you've done that save the route table and then next select the route table in availability zone C so select RT hyphen private C make sure the routes tab is selected and click on edit routes again we'll be adding a route it will be the IP version for default route so 0.0.0.0/0 select a target go to NAT gateway and pick the NAT gateway in availability zone C so NAT GW hyphen C once you've done that save the route table and now our private EC2 instance should be able to ping 1.1.1.1 because we have the routing infrastructure in place so let's move back to our private instance and we can see that it's not actually working now the reason for this is that although we have created these routes we haven't actually associated these route tables with any of the subnets subnets in a VPC which don't have an explicit route table association are associated with the main route table now we need to explicitly associate each of these route tables with the subnets inside that same AZ so let's go ahead and pick RT hyphen private A we'll go through in order so select it click on the subnet associations tab and edit subnet associations and then you need to pick all of the private subnets in AZ A so that's the reserved subnet so reserved hyphen A the app subnet so app hyphen A and the DB subnet so DB hyphen A so all of these are the private subnets in availability zone A notice how all the public subnets are associated with this custom route table you created earlier but the ones we're setting up now are still associated with the main route table so we're going to resolve that now by associating this route table with those subnets so click on save and this will associate all of the private subnets in AZ A with the AZ A route table so now we're going to do the same process for AZ B and AZ C and we'll start with AZ B so select the private B route table click on subnet associations edit subnet associations so select application B database B and then reserved B and then scroll down and save the associations and then select the private C route table click on subnet associations edit subnet associations and then select reserved C database C and then application C and then scroll down and save those associations and now that we've associated these route tables with the subnets and now that we've added those default routes if we go back to session manager where we still have the connection open to the private EC2 instance we should see that the ping has started to work and that's because we now have a NAT gateway providing service to each of the private subnets in all of the three availability zones okay so that's everything you needed to cover in this demo lesson now it's time to clean up the account and return it to the same state as it was at the start of this demo lesson from this point on within the course you're going to be using automation and so we can remove all the configuration that we've done inside this demo lesson so the first thing we need to do is to reverse the route table changes that we've done so we need to go ahead and select the RT hyphen private a route table go ahead and select subnet associations and then edit the subnet associations and then just uncheck all of these subnets and this will return these to being associated with the main route table so scroll down and click on save do the same for RT hyphen private be so deselect all of these associations and click on save and then the same for RT hyphen private see so select it go to subnet associations and then edit them and remove all of these subnets and click on save next select all of these private route tables these are the ones that we created in this lesson so select them all click on the actions drop down and then delete route table and confirm by clicking delete route tables go to NAT gateways on the left and we need to select each of the NAT gateways in turn so a and then click on actions and delete NAT gateway type delete click delete then select be and do the same process actions delete NAT gateway type delete click delete and finally the same for see so select the C NAT gateway click on actions and delete NAT gateway you'll need to type delete to confirm click on delete now we're going to need all of these to be in a fully deleted state before we can continue so hit refresh and make sure that all three NAT gateways are deleted if yours aren't deleted if they're still listed in a deleting state then go ahead and pause the video and resume once all of these have changed to deleted at this point all of the NAT gateways have deleted so you can go ahead and click on elastic IPs and we need to release each of these IPs so select one of them and then click on actions and release elastic IP addresses and click release and do the same process for the other two click on release then finally actions release IP click on release once that's done move back to the cloud formation console select the stack which was created by the one click deployment at the start of the lesson and click on delete and then confirm that deletion and that will remove the cloud formation stack and any resources created as part of this demo and at that point once that finishes deleting the account has been returned into the same state as it was at the start of this demo lesson so I hope this demo lesson has been useful just to reiterate what you've done you've created three NAT gateways for a region resilient design you've created three route tables one in each availability zone added a default IP version for route pointing at the corresponding NAT gateway and associated each of those route tables with the private subnets in those availability zones so you've implemented a regionally resilient NAT gateway architecture so that's a great job that's a pretty complex demo but it's going to be functionality that will be really useful if you're using AWS in the real world or if you have to answer any exam questions on NAT gateways with that being said at this point you have cleared up the account you've deleted all the resources so go ahead complete this video and when you're ready I'll see you in the next.

    1. Hello there, folks.

      Thanks once again for joining.

      Now that we've got a little bit of an understanding of what problem cloud is solving, let's actually go ahead and define it.

      So what we'll talk about is technology on tap, a common phrase that you might have heard about when talking about cloud.

      What is it and why would we say that?

      Then what we're actually going to do is walk through the NIST definition of cloud.

      So there are five key properties that the National Institute of Standards and Technology does use to determine whether or not something is cloud.

      So we'll walk through that.

      So we've got a good understanding of what cloud is and what cloud is not.

      So first things first, technology on tap.

      Why would we refer to cloud as technology on tap?

      Well, let's have a think about the taps we do know about.

      When you want access to water, if you're lucky enough to have access to a nice and easy supply of water, all you really need to do is turn on your tap and get access to as little or as much water as you want.

      You can turn that on and off as you require.

      Now, we know that that's easy for us.

      All we have to worry about is the tap and paying the bill for the amount of water that we consume.

      But what we don't really have to worry about is everything that goes in behind the scenes.

      So the treatment of the water to bring it up to drinking standards, the actual storage of that treated water, and then the transportation of that through the piping network to actually get to our tap.

      All of that is managed for us.

      We don't need to really worry about what happens behind the scenes.

      All we do is focus on that tap.

      We turn it on if we want more.

      We turn it off when we are finished.

      We only pay for what we consume.

      So you might be able to see where I'm going with this.

      This is exactly what we are talking about with cloud.

      With cloud, however, it's not water that we're getting access to, it is technology.

      So if we want access to technology, we use the cloud.

      We push some buttons, we click on an interface, we use whatever tool we require, and we get access to those servers, that storage, that database, whatever it might be that we require in the cloud.

      Now again, behind the scenes, we don't have to worry about the data centers that host all of this technology, all of these services that we want access to.

      We don't worry about the physical infrastructure, the hosting infrastructure, the storage, all the different bits and pieces that actually get that technology to us, we don't need to worry about.

      And how does it get to us?

      How is it available all across the globe?

      Well, we don't need to worry about that connectivity and delivery as well.

      All of this behind the scenes when we use cloud is managed for us.

      All we have to worry about is turning on or off services as we require.

      And this is why you can hear cloud being referred to as technology on tap, because it is very similar to the water utility service.

      Utility service is another name you might hear cloud being referred to, because it's like water or electricity.

      Cloud are like these utility services where you don't have to worry about all the infrastructure behind the scenes.

      You just worry about the thing that you want access to.

      And really importantly, you only have to pay for what you use.

      You turn it on if you need it, you turn it on if you don't, you create things when you need them, delete them when you don't, and you only pay for those services when you have them, even though they are constantly available at your fingertips.

      Now, compare this to the scenario we walked through earlier.

      Traditionally, we would have to buy all of the infrastructure, have it sitting there idly, even if we weren't using it, we would still have had to pay for it, set it up, power it and keep it all running.

      So this is a high level of what we are talking about with cloud.

      Easy access to servers when you need them, turn them off when you don't, don't worry about all that infrastructure behind the scenes.

      But that's a high level definition.

      So let's now walk through what the NIST use as the key properties to define cloud.

      One of the first properties you can use to understand whether something is or is not cloud is understanding whether or not it provides you on demand self service access, where you can easily go ahead and get that technology without even having to talk to humans.

      So what do I really mean by that?

      Well, let's say you're a cloud administrator, you want to go ahead and access some resources in the cloud.

      Now, if you do want access to some services, some data, some storage and application, whatever it might be, while you're probably going to have some sort of admin interface that you can use, whether that's a command line tool or some sort of graphical user interface, you can easily use that to turn on any of the services that you need, web applications, data, storage, compute and much, much more.

      And you don't have to go ahead, talk to another human, procure all of the infrastructure that runs behind the scenes.

      You use your tool, it is self service, it is on demand, create it when you want it, delete it when you don't.

      So that's on demand self service access and one of the key properties of the cloud.

      Next, what I want to talk to you about is broad network access.

      Now, this is where we're just saying, if something is cloud, it should be easy for you to access through standard capabilities.

      So for example, if we are the cloud administrator, it's pretty common when you're working with technology to expect that you would have command line tools, web based tools and so on.

      But even when we're not talking about cloud administrators and we're actually talking about the end users, maybe for example, accessing storage, it should be easy for them to do so through standard tools as well, such as a desktop application, a web browser or something similar.

      Or maybe you've gone ahead and deployed a reporting solution in the cloud, like we spoke of in the previous lesson.

      Well, you would commonly expect for that sort of solution that maybe there's also a mobile application to go and access all of that reporting data.

      The key point here is that if you are using cloud, it is expected that all of the common standard sorts of accessibility options are available to you, public access, private access, desktop applications, mobile applications and so on.

      So if that's what cloud is and how we access it, where actually is it?

      That's a really important part of the definition of cloud.

      And that's where we're referring to resource pooling, this idea that you don't really know exactly where the cloud is that you are going to access.

      So let's say for example, you've got your Aussie Mart company.

      If they want to deploy their solution to be available across the globe, well, it should be pretty easy for them to actually go ahead and do that.

      Now, we don't know necessarily where that is.

      We can get access to it.

      We might say, I want my solution available in Australia East for example, or Europe or India or maybe central US for example.

      All of these refer to general locations where we want to deploy our services.

      When you use cloud, you are not going to go ahead and say, I want one server and I want it deployed to the data center at 123 data center street.

      Okay, you don't know the physical address exactly or at least you shouldn't really have to.

      All you need to know about is generally where you are going to go and deploy that.

      Now, you will also see that for most cloud providers, you've got that global access in terms of all the different locations you can deploy to.

      And really importantly, in terms of all of these pooled resources, understand that it's not just for you to use.

      There will be other customers all across the globe who are using that as well.

      So when you're using cloud, there are lots of resources.

      They might be in lots of different physical locations and lots of different physical infrastructure and in use by lots of different customers.

      And you don't really need to worry about that or know too much about it.

      Another really important property of the cloud is something referred to as rapid elasticity.

      Now elasticity is the idea that you can easily get access to more or less resources.

      And when you work with cloud, you're actually going to commonly hear this being referred to as scaling out and in rather than just scaling up and down.

      So what do I mean by that?

      Well, let's say we've got our users that need to access our Aussie Mart store.

      We might decide to use cloud to host our Aussie Mart web application.

      And perhaps that's hosted on a server and a database.

      Now, when that application gets really busy, for example, if we have lots of different users going to access it at the same time, we might want to scale out to meet demand.

      That is to say, rather than having one server that hosts our web application, we might actually have three.

      And if that demand for our application decreases, we might actually go ahead and decrease the underlying resources that power it as well.

      What we are talking about here is scaling in and out by adding or decreasing the number of resources that host our application.

      This is different from the traditional approach to scalability, where what we would normally do is just add CPU or add memory, for example.

      We would increase the size of one individual resource that was hosting our solution.

      So that's just elasticity at a high level and it's a really key property of cloud.

      Now, we'll just say here that if you are worried about how that actually works behind the scenes in terms of how you host that application across duplicate resources, how you provide connectivity to that, that's all outside the scope of this beginners course, but it's definitely covered in other content as well.

      So when you're using cloud, you get easy access to scale in and out and you should never feel like there are not enough resources to meet your demand.

      To you, it should just feel like if you want a hundred servers, for example, then you can easily get a hundred servers.

      All right, now the last property of cloud that I want to talk to you about is that of measuring service.

      When we're talking about measuring service, what we're talking about is the idea that if you are using cloud to host your solutions, it should be really easy for you to go and say, I know what this is costing, I know where my resources are, how they are performing and whether there are any issues and I can control the types of resources and the configuration that I use that I'm going to deploy.

      So for example, it should be easy for you to say, how much is it going to cost me for five gigabytes of storage?

      What does my bill look like currently and what am I forecasted to be using over the remainder of the month?

      Or maybe you want to say that certain services should not be allowed to be deployed across all regions.

      Yes, cloud can be accessed across the globe, but maybe your organization only works in one part of a specific country and that's the only location you should be able to use.

      These are the standard notions of measuring and controlling service and it's really common to all of the cloud providers.

      All right, everybody.

      So now you've got an understanding of what cloud is and how you can define it.

      If you'd like to see more about this definition from the NIST, then be sure to check out the link that I've included for this lesson.

      So thanks for joining me, folks.

      I'll see you in the next lesson.

    1. Hey there everybody, thanks for joining.

      It's great to have you with me in this lesson where we're going to talk about why cloud matters.

      Now to help answer that question, what I want to do firstly is talk to you about the traditional IT infrastructure.

      How did we used to do things?

      What sort of challenges and issues did we face?

      And therefore we'll get a better understanding of what cloud is actually doing to help.

      We can look at how things used to be and how things are now.

      So what we're going to do throughout this lesson is walk through a little bit of a scenario with a fictitious company called Ozzymart.

      So let's go ahead now, jump in and have a chat about the issues that they're currently facing.

      Ozzymart is a fictitious company that works across the globe selling a range of different Australia related paraphernalia.

      Maybe stuffed toys for kangaroos, koalas and that sort of thing.

      Now they've currently got several different applications that they use that they provide access to for their users.

      And currently the Ozzymart team do not use the cloud.

      So when we have a look at the infrastructure hosting these applications, we'll learn that Ozzymart have a couple of servers, one server for each of the applications that they've got configured.

      Now the Ozzymart IT team have had to have gone and set up these servers with windows, the applications and all the different data that they need for these applications to work.

      And what it's also important to understand about the Ozzymart infrastructure is all of this is currently hosted on their on-premises customer managed infrastructure.

      So yes, the Ozzymart team could have gone out and maybe used a data center provider.

      But the key point here is that the Ozzymart IT team have had to set up servers, operating systems, applications and a range of other infrastructure to support all of this storage, networking, power, cooling.

      Okay, these are the sorts of things that we have to manage traditionally before we were able to use cloud.

      Now to help understand what sort of challenges that might introduce, let's walk through a scenario.

      We're going to say that the Ozzymart CEO has gone and identified the need for reporting to be performed across these two applications.

      And the CEO wants those reports to be up and ready by the end of this month.

      Let's say that's only a week away.

      So the CEO has instructed the finance manager and the finance manager has said, "Hey, awesome.

      You know what?

      I've found this great app out there on the internet called Reports For You.

      We can buy it, download it and install it.

      I'm going to go tell the IT team to get this up and running straight away."

      So this might sound a little bit familiar to some of you who have worked in traditional IT where sometimes demands can come from the top of the organization and they filter down with really tight timelines.

      So let's say for example, the finance manager is going to go along, talk to the IT team and say, "We need this Reports For You application set up by the end of month."

      Now the IT team might be a little bit scared because, hey, when we look at the infrastructure we've got, it's supporting those two servers and applications okay, but maybe we don't have much more space.

      Maybe we don't have enough storage.

      Maybe we are using something like virtualization.

      So we might not need to buy a brand new physical server and we can run up a virtual Windows server for the Reports For You application.

      But there might just not be enough resources in general.

      CPU, memory, storage, whatever it might be to be able to meet the demands of this Reports For You application.

      But you've got a timeline.

      So you go ahead, you get that server up and running.

      You install the applications, the operating system data, all there as quickly as you can to meet these timelines that you've been given by the finance manager.

      Now maybe it's not the best server that you've ever built.

      It might be a little bit rushed and a little bit squished, but you've managed to get that server up and running with the Reports For You application and you've been able to meet those timelines and provide access to your users.

      Now let's say that you've given access to your users for this Reports For You application.

      Now let's say when they start that monthly reporting job, the Reports For You application needs to talk to the data across your other two applications, the Aussie Mart Store and the Aussie Mart Comply application.

      And it's going to use that data to perform the reporting that the CEO has requested.

      So you kick off this report job on a Friday.

      You hope that it's going to be complete on a Saturday, but maybe it's not.

      You check again on a Sunday and things are starting to get a little bit scary.

      And uh-oh, Monday rolls around, the Reports For You report is still running.

      It has not yet complete.

      And that might not be so great because you don't have a lot of resources on-premises.

      And now all of your applications are starting to perform really poorly.

      So that Reports For You application is still running.

      It's still trying to read data from those other two applications.

      And maybe they're getting really, really slow and let's hope not, but maybe the applications even go off entirely.

      Now those users are going to become pretty angry.

      You're going to get a lot of calls to the help desk saying that things are offline.

      And you're probably going to have the finance manager and every other manager reaching out to you saying, this needs to be fixed now.

      So let's say you managed to push through, perhaps through the rest of Monday, and that report finally finishes.

      You clearly need more resources to be able to run this report much more quickly at the end of each month so that you don't have angry users.

      So what are you going to do to fix this for the next month when you need to run the report again?

      Well, you might have a think about ordering some new software and hardware because you clearly don't have enough hardware on-premises right now.

      You're going to have to wait some time for all of that to be delivered.

      And then you're going to have to physically and store it, set it up, get it running, and make sure that you've got everything you need for reports for you to be running with more CPU and resources next time.

      There's a lot of different work that you need to do.

      This is one of the traditional IT challenges that we might face when the business has demands and expectations for things to happen quickly.

      And it's not really necessarily the CEO or the finance manager's fault.

      They are focused on what the business needs.

      And when you work in the technology teams, you need to do what you can to support them so that the business can succeed.

      So how might we do that a little bit differently with cloud?

      Well, with cloud, we could sign up for a cloud provider, we could turn on and off servers as needed, and we could scale up and scale down, scale in and scale out resources, all to meet those demands on a monthly basis.

      So that could be a lot less work to do and it could certainly provide you the ability to respond much more quickly to the demands that come from the business.

      And rather than having to go out and buy all of this new infrastructure that you are only going to use once a month, well, as we're going to learn throughout this course, one of the many benefits of cloud is that you can turn things on and off really quickly and only pay for what you need.

      So what might this look like with cloud?

      Well, with cloud, what we might do is no longer have that on-premises rushed server that we were using for reports for you.

      Instead of that, we can go out to a public cloud provider like AWS, GCP or hopefully Azure, and you can set up those servers once again using a range of different features, products that are all available through the various public cloud providers.

      Now, yes, in this scenario, we are still talking about setting up a server.

      So that is going to take you some time to configure Windows, set up the application, all of the data and configuration that you require, but at least you don't need to worry about the actual physical infrastructure that is supporting that server.

      You don't have to go out, talk to your procurement team, talk to a different providers, wait for different physical infrastructure to be delivered and licensing and software and other assets.

      With cloud, as we will learn, you can really quickly get online and up and running.

      And also, if we had that need to ensure that the reports for you application was running with lots of different resources at the end of the month, it's much easier when we use cloud to just go and turn some servers on and then maybe turn them off at the end of the month when they are no longer acquired.

      This is the sort of thing that we are talking about with cloud.

      We're only really just touching on the service about what cloud can do and what cloud actually is.

      But my hope is that through this lesson, you can understand how cloud changes things.

      Cloud allows us to work with technology in a much different way than we traditionally would work with our on-premises infrastructure.

      Another example that shows how cloud is different is that rather than using the reports for you application, what we might in fact actually choose to do is go to a public cloud provider and go to someone that actually has a equivalent reports for you solution that's entirely built in the cloud ready to go.

      In this way, not only do we no longer have to manage the underlying physical infrastructure, we don't actually have to manage the application software installation, configuration, and all of that service setup.

      With something like a reporting software that's built in the cloud, we would just provide access to our users and only have to pay on a per user basis.

      So if you've used something like zoom for meetings or Dropbox for data sharing, that's the sort of solution we're talking about.

      So if we consider this scenario for Aussie Mart, we have a think about the benefits that they might access when they use the cloud.

      Well, we can much more quickly get access to resources to respond to demand.

      If we need to have a lot of different compute capacity working at the end of the month with cloud, like you'll learn, we can easily get access to that.

      If we wanted to add lots of users, we could do that much more simply as well.

      And something that the finance manager might really be happy about in this scenario is that we aren't going to go back and suggest to them that we need to buy a whole heap of new physical infrastructure right now.

      When we think about traditionally how Aussie Mart would have worked with this scenario, they would have to go and buy some new physical servers, resources, storage, networking, whatever that might be, to meet the needs of this reports for you application.

      And really, they're probably going to have to strike a balance between having enough infrastructure to ensure that the reports for you application completes its job quickly and not buying too much infrastructure that's just going to be sitting there unused whilst the reports for you application is not working.

      And really importantly, when we go to cloud, we see this difference as not having to buy lots of physical infrastructure upfront as being referred to as capital expenditure versus operational expenditure.

      Really, what we're just saying here is rather than spending a whole big lump sum all at once to get what you need, you can just pay on a monthly basis for what you need when you need it.

      And finally, one of the other benefits that you'll also see is that we're getting a reduction in the amount of different tasks that we have to complete in terms of IT administration, set up of operating systems, management of physical infrastructure, what the procurement team has to manage, and so on.

      Again, right now we're just talking really high level about a fictitious scenario for Aussie Mart to help you to understand the types of things and the types of benefits that we can get access to for cloud.

      So hopefully if you're embarking on a cloud journey, you're gonna have a happy finance manager, CEO, and other team members that you're working with as well.

      Okay, everybody, so that's a wrap to this lesson on why cloud matters.

      As I've said, we're really only just scratching the surface.

      This is just to introduce you to a scenario that can help you to understand the types of benefits we get access to with cloud.

      As we move throughout this course, we'll progressively dive deeper in terms of what cloud is, how you define it, the features you get access to, and other common concepts and terms.

      So thanks for joining me, I'll see you there.

    1. Welcome back and in this brief demo lesson I want to give you some experience of working with both EC2 instance connect as well as connecting with a local SSH client.

      Now these are both methods which are used for connecting to EC2 instances both with public IP version 4 addressing and IP version 6 addressing.

      Now to get started we're going to need some infrastructure so make sure that you're logged in as the IAM admin user into the general AWS account which is the management account of the organization and as always you'll need the northern Virginia region selected.

      Now in this demonstration you are going to be connecting to an EC2 instance using both instance connect and a local SSH client and to use a local SSH client you need a key pair.

      So to create that let's move across to the EC2 console, scroll down on the left and select key pairs.

      Now you might already have key pairs created from earlier in the course.

      If you have one created which is called A4L which stands for Animals for Life then that's fine.

      If you don't we're going to go ahead and create that one.

      So click on create key pair and then under name we're going to use A4L.

      Now if you're using Windows 10 or Mac OS or Linux then you can select the PEM file format.

      If you're using Windows 8 or prior then you might need to use the putty application and to do that you need to select PPK.

      But for this demonstration I'm going to assume that you're using the PEM format.

      So again this is valid on Linux, Mac OS or any recent versions of Microsoft Windows.

      So select PEM and then click on create key pair and when you do it's going to present you with a download.

      It's going to want you to save this key pair to your local machine so go ahead and do that.

      Once you've done that from the AWS console attached to this lesson is a one-click deployment link.

      So I want you to go ahead and click that link.

      That's going to move you to a quick create stack screen.

      Everything should be pre-populated.

      The stack name should be EC2 instance connect versus SSH.

      The key name box should already be pre-populated with A4L which is a key that you just created or one which you already had.

      Just move down to the very bottom, check the capabilities box and then click on create stack.

      Now you're going to need this to be in a create complete state before you continue with the demo lesson.

      So pause the video, wait for your stack to change to create complete and then you're good to continue.

      Okay so this stacks now in a create complete status and we're good to continue.

      Now if we click on the resources tab you'll see that this has created the standard animals for life VPC and then it's also created a public EC2 instance.

      So this is an EC2 instance with a public IP version 4 address that we can use to connect to.

      So that's what we're going to do.

      So click on services and then select EC2 to move to the EC2 console.

      Once you're there click on instances running and you should have a single EC2 instance A4L-publicEC2.

      Now the two different ways which I want to demonstrate connecting to this instance in this demo lesson are using a local SSH client and key based authentication and then using the EC2 instance connect method.

      And I want to show you how those differ and give you a few hints and tips which might come in useful for production usage and for the exams.

      So if we just go ahead and select this instance and then click on the security tab you'll see that we have this single security group which is associated to this instance.

      Now make sure the inbound rules is expanded and just have a look at what network traffic is allowed by this security group.

      So the first line allows port 80 TCP which is HTTP and it allows that to connect to the instance from any source IP address specifically IP version 4.

      We can tell it's IP version 4 because it's 0.0.0.0/0 which represents any IP version 4 address.

      Next we allow port 22 using TCP and again using the IP version 4 any IP match and this is the entry which allows SSH to connect into this instance using IP version 4.

      And then lastly we have a corresponding line which allows SSH using IP version 6.

      So we're allowing any IP address to connect using SSH to this EC2 instance.

      And so connecting to it using SSH is relatively simple.

      We can right click on this instance and select connect and then choose SSH client and AWS provides us with all of the relevant information.

      Now note how under step number three we have this line which is chmod space 400 space a4l.pm.

      I want to demonstrate what happens if we attempt to connect without changing the permissions on this key file.

      So to do that right at the bottom is an example command to connect to this instance.

      So just copy that into your clipboard.

      Then I want you to move to your command prompt or terminal.

      In my case I'm running macOS so I'm using a terminal application.

      Then you'll need to move to the folder where you have the PEM file stored or where you just downloaded it in one of the previous steps.

      I'm going to paste in that command which I just copied onto my clipboard.

      This is going to use the a4l.pm file as the identity information and then it's going to connect to the instance using the EC2-user local Linux user.

      And this is the host name that it's going to connect to.

      So this is my EC2 instance.

      Now I'm going to press enter and attempt that connection.

      First it will ask me to verify the authenticity of this server.

      So this is an added security method.

      This is getting the fingerprint of this EC2 instance.

      And it means that if we independently have a copy of this fingerprint, say from the administrator of the server that we're connecting to, then we can verify that we're connecting to that same server.

      Because it's possible that somebody could exploit DNS and replace a legitimate DNS name with one which points at a non-legitimate server.

      So that's important.

      You can't always rely on a DNS name.

      DNS names can be adjusted to point at different IP addresses.

      So this fingerprint is a method that you can use to verify that you're actually connecting to the machine or the instance which you think you are.

      Now in this case, because we've just created this EC2 instance, we can be relatively certain that it is valid.

      So we're just going to go ahead and type yes and press enter.

      And then it will try to connect to this instance.

      Now immediately in my case, I got an error.

      And this error is going to be similar if you're using macOS or Linux.

      If you're using Windows, then there is a chance that you will get this error or won't.

      And if you do get it, it might look slightly different.

      But look for the keyword of permissions.

      If you see that you have a permissions problem with your key, then that's the same error as I'm showing on my screen now.

      Basically what this means is that the SSH client likes it when the permissions on these keys are restricted, restricted to only the user that they belong to.

      Now in my case, the permissions on this file are 644.

      And this represents my user, my group, and then everybody.

      So this means this key is accessible to other users on my local system.

      And that's far too open to be safe when using local SSH.

      Now in Windows, you might have a similar situation where other users of your local machine have read permissions on this file.

      What this error is telling us to do is to correct those permissions.

      So if we go back to the AWS console, this is the command that we need to run to correct those permissions.

      So copy that into your clipboard, move back to your terminal, paste that in, and press enter.

      And that will correct those permissions.

      Now under Windows, the process is that you need to edit the permissions of that file.

      So right click properties and then edit the security.

      And you need to remove any user access to that file other than your local user.

      And that's the same process that we've just done here, only in Windows it's GUI based.

      And under Mac OS or Linux, you use CHmod.

      So now that we've adjusted those permissions, if I use the up arrow to go back to the previous command and press enter, I'm able to connect to the CC2 instance.

      And that's using the SSH client.

      To use the SSH client, you need to have network connectivity to the CC2 instance.

      And you need to have a valid SSH key pair.

      So you need the key stored on your local machine.

      Now this can present scalability issues because if you need to have a large team having access to this instance, then everybody in that team need a copy of this key.

      And so that does present admin problems if you're doing it at scale.

      Now in addition to this, because you're connecting using an SSH client from your local machine, you need to make sure that the security group of this instance allows connections from your local machines.

      So in this case, it allows connections from any source IP address into this instance.

      And so that's valid for my IP address.

      You need to make sure that the security group on whichever instance you're attempting to connect to allows your IP address as a minimum.

      Now another method that you can use to connect to EC2 is EC2 instance connect.

      Now to use that, we right click, we select connect, and we have a number of options at the top.

      One of these is the SSH client that we've just used.

      Another one is EC2 instance connect.

      So if we select this option, we're able to connect to this instance.

      It shows us the instance ID, it shows us the public IP address, and it shows us the user to connect into this instance with.

      Now AWS attempt to automatically determine the correct user to use.

      So when you launch an instance using one of the default AMIs, then it tends to pick correctly.

      However, if you generate your own custom AMI, it often doesn't guess correctly.

      And so you need to make sure that you're using the correct username when connecting using this method.

      But once you've got the correct username, you can just go ahead and click on connect, and then it will open a connection to that instance using your web browser.

      It'll take a few moments to connect, but once it has connected, you'll be placed at the terminal of this EC2 instance in exactly the same way as you were when using your local SSH.

      Now one difference you might have noticed is at no point where you prompted to provide a key.

      When you're using EC2 instance connect, you're using AWS permissions to connect into this instance.

      So because we're logged in using an admin user, we have those permissions, but you do need relevant permissions added to the identity of whoever is using instance connect to be able to connect into the instance.

      So this is managed using identity policies on the user, the group or the role, which is attempting to access this instance.

      Now one important element of this, which I want to demonstrate, if we go back to instances and we select the instance, click on security, and then click on the security group, which is associated with this instance.

      Scroll down, click on edit inbound rules, and then I want you to locate the inbound rule for IP version 4 SSH, SSH TCP 22, and then it's using this catchall, so 0.0.0.0/0, which represents any IP version 4 address.

      So go ahead and click on the cross to remove that, and then on that same line in the source area, click on this drop down and change it to my IP.

      So this is my IP address, yours will be different, but then we're going to go ahead and save that rule.

      Now just close down the tab that you've got connected to instance connect, move back to the terminal, and type exit to disconnect from that instance, and then just rerun the previous command.

      So connect back to that instance using your local SSH client.

      You'll find that it does reconnect because logically enough, this connection is coming from your local IP address, and you've changed the security group to allow connections from that address, so it makes sense that this connection still works.

      Moving back to the console though, let's go to the EC2 dashboard, go to running instances, right click on this instance, go to connect, select EC2 instance connect, and then click on connect and just observe what happens.

      Now you might have spent a few minutes waiting for this to connect, and you'll note that it doesn't connect.

      Now this might seem strange at this point because you're connecting from a web browser, which is running on your local machine.

      So it makes sense that if you can connect from your local SSH client, which is also running on your local machine, you should be able to connect using EC2 instance connect.

      Now this might seem logical, but the crucial thing about EC2 instance connect is that it's not actually originating connections from your local machine.

      What's happening is that you're making a connection through to AWS, and then once your connection arrives at AWS, the EC2 instance connect service is then connecting to the EC2 instance.

      Now what you've just done is you've edited the security group of this instance to only allow your local IP address to connect, and this means that the EC2 instance connect service can no longer connect to this instance.

      So what you need in order to allow the EC2 instance connect service to work is you either need to allow every source IP address, so 0.0.0.0.0/0, but of course that's bad practice for production usage.

      It's much more secure if you go to this URL, and I'll make sure that I include this attached to this lesson.

      This is a list of all of the different IP ranges which AWS use for their services.

      Now because I have this open in Firefox, it might look a little bit different.

      If I just go to raw data, that might look the same as your browser.

      If you're using Firefox, you have the ability to open this as a JSON document.

      Both of them show the same data, but when it's JSON, you have the ability to collapse these individual components.

      But the main point about this document is that this contains a list of all of the different IP addresses which are used in each different region for each different service.

      So if we wanted to allow EC2 instance connect for a particular region, then we might search for instance, locate any of these items which have EC2 instance connect as the service, and then just move through them looking for the one which matches the region that we're using.

      Now in my case, I'm using US East One, so I'd scroll through all of these IP address ranges looking for US East One.

      There we go, I've located it.

      It's using this IP address range.

      So I might copy this into my clipboard, move back to the EC2 console, select the instance, click on security, select the security group of this instance, scroll down, edit the inbound rules, remove the entry for my IP address, paste in the entry for the EC2 instance connect service, and then save that rule.

      And now what you'll find if you move back to your terminal and try to interact with this instance, you might be able to initially because the connection is still established, but if you exit and then attempt to reconnect, this time you'll see that you won't be able to connect because now your local IP address is no longer allowed to connect to this instance.

      However, if you move back to the AWS console, go to the dashboard and then instance is running, right click on the instance and put connect, select instance connect and then click on connect.

      Now you'll be allowed to connect using EC2 instance connect.

      And the reason for that just to reiterate is that you've just edited the security group of this EC2 instance and you've allowed the IP address range of the EC2 instance connect service.

      So now you can connect to this instance and you could do so at scale using AWS permissions.

      So I just wanted to demonstrate how both of those connection methods work, both instance connect and using a local SSH client.

      That's everything I wanted to cover.

      So just go ahead and move back to the CloudFormation console, select this stack that you created using the one click deployment, click on delete and then confirm that process.

      And that will clear up all of the infrastructure that you've used in this demo lesson.

      At this point though, that's everything I wanted to cover.

      So go ahead, complete this video and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      So focusing specifically on the animals for life scenario.

      So what we're going to do in the upcoming demo lesson, to implement a truly resilient architecture for net services in a VPC, you need a net gateway in a public subnet inside each availability zone that the VPC uses.

      So just like on the diagram that you've gone through now.

      And then as a minimum, you need private route tables in each availability zone.

      In this example, AZA, AZB, and then AZC.

      Each of these would need to have their own route table, which would have a default IP version for route, which points at the net gateway in the same availability zone.

      That way, if any availability zone fails, the others could continue operating without issues.

      Now, this is important.

      I've seen it in a few of some questions.

      Where it suggests that one net gateway is enough, that a net gateway is truly regionally resilient.

      This is false.

      A net gateway is highly available in the availability zone that it's in.

      So if hardware fails or it needs to scale to cope with load, it can do so in that AZ.

      But if the whole AZ fails, there is no failover.

      You provision a net gateway into a specific availability zone, not the region.

      It's not like the internet gateway, which by default is region resilient.

      For a net gateway, you have to deploy one into each AZ that you use if you need that region resilience.

      Now, my apologies in advance for the small text.

      It's far easier to have this all on screen at once.

      I mentioned at the start of the lesson that net used to be provided by net instances, and these are just for the net process running on an EC2 instance.

      Now, I don't expect this to feature on the exam at this point.

      But if you ever need to use a net instance, by default, EC2 filters all traffic that it sends or receives.

      It essentially drops any data that is on its network card when that network card is not either the source or the destination.

      So if an instance is running as a net instance, then it will be receiving some data which the source address will be of other resources in that VPC.

      And the destination will be a host on the internet.

      So it will neither be the source nor the destination.

      So by default, that traffic will be dropped.

      And if you need to allow an EC2 instance to function as a net instance, then you need to disable a feature called source and destination checks.

      This can be disabled via the console UI, the CLI, or the API.

      The only reason I mention this is I have seen this question in the exam before, and if you do implement this in a real-world production-style scenario, you need to be aware that this feature exists.

      I don't want you wasting your time trying to diagnose this feature.

      So if you just right-click on an instance in the console, you'll be able to see an option to disable source and destination checks.

      And that is required if you want to use an EC2 instance as a net instance.

      Now, at the highest level, architecturally, net instances and net dayways are kind of the same.

      They both need a public ID address.

      They both need to run in a public subnet, and they both need a functional internet gateway.

      But at this point, it's not really preferred to use EC2 running as a net instance.

      It's much easier to use a net gateway, and it's recommended by AWS in most situations.

      But there are a few key scenarios where you might want to consider using an EC2-based net instance.

      So let's just step through some of the criteria that you might be looking at when deploying net services.

      If you value availability, bandwidth, low levels of maintenance, and high performance, then you should use net gateways.

      That goes for both real-world production usage, as well as being default for answering any exam questions.

      A net gateway offers high-end performance, its scales, its custom design, perform network address translation.

      A net instance in comparison is limited by the capabilities of the instances running on, and that instance is also general purpose, so it won't offer the same level of custom design performance as a net gateway.

      Now, availability is another important consideration, and that instance is a single EC2 instance running inside an availability zone.

      It will fail if the EC2 hardware fails.

      It will fail if its storage fails or if its network fails, and it will fail if the AZ itself fails entirely.

      A net gateway has some benefits over a net instance.

      So inside one availability zone, it's highly available, so it can automatically recover, it can automatically scale.

      So it removes almost all of the risks of outage versus a net instance.

      But it will still fail entirely if the AZ fails entirely.

      You still need provision, multiple net gateways, spread across all the AZs that you intend to use, if you want to ensure complete availability.

      For maximum availability, a net gateway in every AZ you use.

      This is critical to remember for the exam.

      Now, if cost is your primary choice, if you're a financially challenged business, or if the VPC that you're deploying net services into is just a test VPC or something that's incredibly low volume, then a net instance can be cheaper.

      It can also be significantly cheaper at high volumes of data.

      You've got a couple of options.

      You can use a very small EC2 instance, even ones that are free tier eligible to reduce costs, and the instances can also be fixed in size, meaning they offer predictable costs.

      A net gateway will scale automatically, and you'll build for both the net gateway and the amount of data transferred, which increases as the gateway scales.

      A net gateway is also not free tier eligible.

      Now, this is really important because when we deploy these in the next demo lesson, it's one of those services that I need to warn you will come at a cost, so you need to be aware of that fact.

      You will be charged for a net gateway regardless of how small the usage.

      Net instances also offer other niche advantages because they're just EC2 instances.

      You can connect to them just like you would any other EC2 instance.

      You can multi-purpose them so you can use them for other things, such as passing hosts.

      You can also use them for port forwarding, so you can have the port on the instance externally that could be connected to over the public internet, and have this forwarded-on for an instance inside the VPC.

      Maybe port 8 if a web, or port 443 for secure web.

      You can be completely flexible when you use net instances.

      With a net gateway, this isn't possible because you don't have access to manage it.

      It's a managed service.

      Now, this comes up all the time in the exam, so try and get it really clear in your memory, and that gateway cannot be used as a passing host.

      It cannot do port forwarding because you cannot connect to its operating system.

      Now, finally, this is again one focus on the exam.

      Net instances are just EC2 instances, so you can filter traffic using the network ACLs on the subnet instances in, or security groups directly associated with that instance.

      Net gateways don't support security groups.

      You can only use knuckles with net gateways.

      This one comes up all the time in the exam, so it's worth noting down and maybe making a flashcard with.

      Now, a few more things before we finish up.

      What about IP version 6?

      The focus of net is to allow private IP version 4 addresses to be used to connect in an outgoing only way to the AWS public zone and public internet.

      Inside AWS, all IP version 6 addresses are publicly routable, so this means that you do not require net when using IP version 6.

      The internet gateway works directly with IP version 6 addresses, so if you choose to make an instance in a private subnet, have a default IP version 6 route to the internet gateway, it will become a public instance.

      As long as you don't have any knuckles or any security groups, any IP version 6 IP address in AWS can communicate directly with the AWS public zone and the public internet.

      So the internet gateway can work directly with IP version 6.

      Net gateways do not work with IP version 6, they're not required and they don't function with IP version 6.

      So for the exam, if you see any questions which mention IP version 6 and net gateways, you can exclude the answer.

      Net gateways do not work with IP version 6 and you can repeat it because I really wanted to stick in your memory.

      So with any subnet inside AWS, which has been configured for IP version 6, if you add the IP version 6 default route, which is colon colon 4 slash 0, if you add that route and you point that route at the internet gateway as a target, that will give that instance bi-directional connectivity to the public internet and it will allow it to reach the AWS public zone and public services.

      One service that we'll be talking about later on in the course when I cover more advanced features of VPC is a different type of gateway, known as an egress-only internet gateway.

      This is a specific type of internet gateway that works only with IP version 6 and you use it when you want to give an IP version 6 instance outgoing only access to the public internet and the AWS public zone.

      So don't worry, we'll be covering that later in the course, but I want to get it really burned into your memory that you do not use net and you do not use net gateways with IP version 6.

      It will not work.

      Now to get you some experience of using net gateways, it's time for a demo.

      In the demo lesson, I'm going to be stepping you through what you need to do to provision a completely resilient net gateway architecture.

      So that's using net gateway in each availability zone as well as configuring the routing required to make it work.

      It's going to be one of the final pieces to our multi-tier VPC and it will allow private instances to have full outgoing internet access.

      Now I can't wait for us to complete this together.

      It's going to be a really interesting demo, one that will be really useful if you're doing this in the real world or if you have to answer exam questions related to net or net gateway.

      So go ahead, complete the video and when you're ready, join me in the demo.

    1. Welcome back.

      In this lesson, I'll be talking about Network Address Translation, or NAT, a process of giving a private resource outgoing only access to the internet.

      And a NAT gateway is the AWS implementation that's available within WPC.

      There's quite a bit of theory to cover, so let's get started.

      So what is NAT?

      Well, it stands for Network Address Translation.

      This is one of those terms which means more than people think that it does.

      In a strict sense, it's a set of different processes which can adjust ID packets by changing their source or destination IP addresses.

      Now, you've seen a form of this already.

      The internet gateway actually performs a type of NAT known as static NAT.

      It's how a resource can be allocated with a public IP version for address, and then when the packets of data leave those resources and pass through the internet gateway, it adjusts the source IP address on the packet from the private address to the public, and then sends the packet on, and then when the packet returns, it adjusts the destination address from the public IP address to the original private address.

      That's called static NAT, and that's how the internet gateway implements public IP version for addressing.

      Now, what most people think of when they think of NAT is a subset of NAT called IP Masquerading.

      And IP Masquerading hides a whole private side IP block behind a single public IP.

      So rather than the one private IP to one public IP process that the internet gateway does, NAT is many private IPs to one single IP.

      And this technique is popular because IP version 4 addresses are running out.

      The public address space is rapidly becoming exhausted.

      IP Masquerading, or what we'll refer to for the rest of this lesson as NAT, gives a whole private range of IP addresses outgoing only access to the public internet and the AWS public zone.

      I've highlighted outgoing because that's the most important part, because many private IPs use a single public IP.

      Incoming access doesn't work.

      Private devices that use NAT can initiate outgoing connections to internet or AWS public space services, and those connections can receive response data, but you cannot initiate connections from the public internet to these private IP addresses when NAT is used.

      It doesn't work that way.

      Now, AWS has two ways that it can provide NAT services.

      Historically, you could use an EC2 instance configured to provide NAT, but it's also a managed service, the NAT gateway, which you can provision in the VPC to provide the same functionality.

      So let's look at how this works architecturally.

      This is a simplified version of the Animals for Life architecture that we've been using so far.

      On the left is an application tier subnet in blue, and it's using the IP range 10.16.32.0/20.

      So this is a private only subnet.

      Inside it are three instances, I01, which is using the IP 10.16.32.10, I02, which is using 32.20, and I03, which is using 32.30.

      These IP addresses are private, so they're not publicly routable.

      They cannot communicate with the public internet or the AWS public zone services.

      These addresses cannot be routed across a public style network.

      Now, if we wanted this to be allowed, if we wanted these instances to perform certain activities using public networking, for example, software updates, how would we do it?

      Well, we could make the subnet's public in the same way that we've done with the public subnets or the web subnets, but we might not want to do that architecturally.

      With this multi-tier architecture that we're implementing together, part of the design logic is to have tiers which aren't public and aren't accessible from the public internet.

      Now, we could also host some kind of software update server inside the VPC, and some businesses choose to do that.

      Some businesses run Windows update services, all Linux update services inside their private network, but that comes with an admin overhead.

      NAT offers us a third option, and it works really well in this style of situation.

      We provision a NAT gateway into a public subnet, and remember, the public subnet allows us to use public IP addresses.

      The public subnet has a route table attached to it, which provides default IP version 4 routes pointing at the internet gateway.

      So, because the NAT gateway is located in this public web subnet, it has a public IP which is routable across the public internet, so it's now able to send data out and get data back in return.

      Now, the private subnet where the instances are located can also have its own route table, and this route table can be different than the public subnet route table.

      So, we could configure it so that the route table that's on the application subnet has a default IP version 4 route, but this time, instead of pointing at the internet gateway, like the web subnet users, we configure this private route table so that it points at the NAT gateway.

      This means when those instances are sending any data to any IP addresses that do not belong inside the VPC, by default, this default route will be used, and that traffic will get sent to the NAT gateway.

      So, let's have a look at how this packet flow works.

      Let's simulate the flow packets from one of the private instances and see what the NAT gateway actually does.

      So, first, instance 1 generates some data.

      Let's assume that it's looking for software updates.

      So, this packet has a source IP address of instance 1's private IP and a destination of 1.3.3.7.

      For this example, let's assume that that's a software update server.

      Now, because we have this default route on the route table of the application subnet, that packet is routed through to the NAT gateway.

      The NAT gateway makes a record of the data packet.

      It stores the destination that the packet is for, the source address of the instance sending it, and other details which help it identify the specific communication in future.

      Remember, multiple instances can be communicating at once, and for each instance, it could be having multiple conversations with different public internet hosts.

      So, the NAT gateway needs to be able to uniquely identify those.

      So, it records the IP addresses involved, the source and destination, the port numbers, everything it needs, into a translation table.

      So, the NAT gateway maintains something called a translation table which records all of this information.

      And then, it adjusts the packet to the one that's been sent by the instance, and it changes the source address of this IP packet to be its own source address.

      Now, if this NAT appliance were anywhere for AWS, what it would do right now is adjust the packet with a public routable address. - Hi. - Let's do this directly.

      But remember, all the inside of the IPC really has directly attached to it a public IP version 4 address.

      That's what the internet gateway does.

      So, the NAT gateway, because it's in the web subnet, it has a default route, and this default route points at the internet gateway.

      And so, the packet is moved from the NAT gateway to the internet gateway by the IPC router.

      At this point, the internet gateway knows that this packet is from the NAT gateway.

      It knows that the NAT gateway has a public IP version 4 address associated with it, and so, it modifies the packet to have a source address of the NAT gateway's public address, and it sends it on its way.

      The NAT gateway's job is to allow multiple private IP addresses to masquerade behind the IP address that it has.

      That's where the term IP masquerading comes from.

      That's why it's more accurate.

      So, the NAT gateway takes all of the incoming packets from all of the instances that it's managing, and it records all the information about the communication.

      It takes those packets, it changes the source address from being those instances to its own IP address, its own external-facing IP address.

      If it was outside AWS, this would be a public address directly.

      That's how your internet router works for your home network.

      All of the devices internally on your network talk out using one external IP address, your home router uses NAT.

      But because it's in AWS, it doesn't have directly attached a real public IP.

      The internet gateway translates from its IP address to the associated public one.

      So, that's how the flow works.

      If you need to give an instance its own public IP version for address, then only the internet gateway is required.

      If you want to give private instances outgoing access to the internet and the AWS public services such as S3, then you need both the NAT gateway to do this many-to-one translation and the internet gateway to translate from the IP of the NAT gateway to a real public IP version for address.

      Now, let's quickly run through some of the key facts for the NAT gateway product that you'll be implementing in the next demo lesson.

      First, and I hope this is logical for you by now, it needs to run from a public subnet because it needs to be able to be assigned a public IP version for IP address for itself.

      So, to deploy a NAT gateway, you already need your VPC in a position where it has public subnets.

      And for that, you need an internet gateway, subnets configured to allocate public IP version for addresses and default routes for those subnets pointing at the internet gateway.

      Now, a NAT gateway actually uses a special type of public IP version for address that we haven't covered yet called an elastic IP.

      For now, just know that these are IP version for addresses, which is static.

      They don't change.

      These IP addresses are allocated to your account in a region and they can be used for whatever you want until you reallocate them.

      And NAT gateways use these elastic IPs, the one service which utilizes elastic IPs.

      Now, they're talking about elastic IPs later on in the course.

      Now, NAT gateways are an AZ resilient service.

      If you read the AWS documentation, you might get the impression that they're fully resilient in a region like an internet gateway.

      They're not, they're resilient in the AZ that they're in.

      So they can recover from hardware failure inside an AZ.

      But if an AZ entirely fails, then the NAT gateway will also fail.

      For a fully region resilient service, so to mirror the high availability provided by an internet gateway, then you need to deploy one NAT gateway in each AZ that you're using in the VPC and then have a route table for private subnets in that availability zone, pointing at the NAT gateway also in that availability zone.

      So for every availability zone that you use, you need one NAT gateway and one route table pointing at that NAT gateway.

      Now, they aren't super expensive, but it can get costly if you have lots of availability zones, which is why it's important to always think about your VPC design.

      Now, NAT gateways are a managed service.

      You deploy them and AWS handle everything else.

      They can scale to 45 gigabits per second in bandwidth and you can always deploy multiple NAT gateways and split your subnets across multiple provision products.

      If you need more bandwidth, you can just deploy more NAT gateways.

      For example, you could split heavy consumers across two different subnets in the same AZ, have two NAT gateways in the same AZ and just route each of those subnets to a different NAT gateway and that would quickly allow you to double your available bandwidth.

      With NAT gateways, you'll build based on the number that you have.

      So there's a standard hourly charge for running a NAT gateway and this is obviously subject to change in a different region, but it's currently about four cents per hour.

      And note, this is actually an hourly charge.

      So partial hours are billed as full hours.

      And there's also a data processing charge.

      So that's the same amount as the hourly charge around four cents currently per gigabyte of processed data.

      So you've got this base charge that a NAT gateway consumes while running plus a charge based on the amount of data that you process.

      So keep both of those things in mind for any NAT gateway related questions in the exam.

      Don't focus on the actual values, just focus on the fact they have two charging elements.

      Okay, so this is the end of part one of this lesson.

      It's getting a little bit on the long side, and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part two will be continuing immediately from the end of part one.

      So go ahead, complete the video, and when you're ready, join me in part two.

    1. Welcome back.

      In this lesson I want to talk in detail about security groups within AWS.

      These are the second type of security filtering feature commonly used within AWS, the other type being network access control lists which we've previously discussed.

      So security groups and knuckles share many broad concepts but the way they operate is very different and it's essential that you understand those differences and the features offered by security groups for both the exam and real-world usage.

      So let's just jump in and get started.

      In the lesson on network access control lists I explained that they're stateless and by now you know what stateless and stateful mean.

      Security groups are stateful, they detect response traffic automatically for a given request and this means that if you allow an inbound or outbound request then the response is automatically allowed.

      You don't have to worry about configuring ephemeral ports, it's all handled by the product.

      If you have a web server operating on TCP 443 and you want to allow access from the public internet then you'll add an inbound security group rule allowing inbound traffic on TCP 443 and the response which is using ephemeral ports is automatically allowed.

      Now security groups do have a major limitation and that's that there is no explicit deny.

      You can use them to allow traffic or you can use them to not allow traffic and this is known as an implicit deny.

      So if you don't explicitly allow traffic then you're implicitly denying it but you can't and this is important you're unable to explicitly deny traffic using security groups and this means that they can't be used to block specific bad actors.

      Imagine you allow all source IP addresses to connect to an instance on port 443 but then you discover a single bad actor is attempting to exploit your web server.

      Well you can't use security groups to block that one specific IP address or that one specific range.

      If you allow an IP or if you allow an IP range or even if you allow all IP addresses then security groups cannot be used to deny a subset of those and that's why typically you'll use network access control lists in conjunction with security groups where the knuckles are used to add explicit denies.

      Now security groups operate above knuckles on the OSI7 layer stack which means that they have more features.

      They support IP and side-based rules but they also allow referencing AWS logical resources.

      This includes all the security groups and even itself within rules.

      I'll be covering exactly how this works on the next few screens.

      Just know at this stage that it enables some really advanced functionality.

      An important thing to understand is that security groups are not attached to instances nor are they attached to subnet.

      They're actually attached to specific elastic network interfaces known as ENIs.

      Now even if you see the user interface present this as being able to attach a security group to an instance know that this isn't what happens.

      When you attach a security group to an instance what it's actually doing is attaching the security group to the primary network interface of that instance.

      So remember security groups are attached to network interfaces that's an important one to remember for the exam.

      Now at this point let's step through some of the unique features of security groups and it's probably better to do this visually.

      Let's start with a public subnet containing an easy to instance and this instance has an attached primary elastic network interface.

      On the right side we have a customer Bob and Bob is accessing the instance using HDTBS so this means TCP but 443.

      Conceptually think of security groups as something which surrounds network interfaces in this case the primary interface of the EC2 instance.

      Now this is how a typical security group might look.

      It has inbound and outbound rules just like a network ACL and this particular example is showing the inbound rules allowing TCP port 443 to connect from any source.

      The security group applies to all traffic which enters or leaves the network interface and because they're stateful in this particular case because we've allowed TCP port 443 as the request portion of the communication the corresponding response part the connection from the instance back to Bob is automatically allowed.

      Now lastly I'm going to repeat this point several times throughout this lesson.

      Security groups cannot explicitly block traffic.

      This means with this example if you're allowing 0.0.0.0.0 to access the instance on port TCP port 443 and this means the whole IP version for internet then you can't block anything specific.

      Imagine Bob is actually a bad actor.

      Well in this situation security groups cannot be used to add protection.

      You can't add an explicit deny for Bob's IP address.

      That's not something that security groups are capable of.

      Okay so that's the basics.

      Now let's look at some of the advanced bits of security group functionality.

      Security groups are capable of using logical references.

      Let's step through how this works with a similar example to the one you just saw.

      We start with a VPC containing a public web subnet and a private application subnet.

      Inside the web subnet is the Categoram application web instance and inside the app subnet is the back-end application instance.

      Both of these are protected by security groups.

      We have A4L-web and A4L-app.

      Traffic wise we have Bob accessing the web instance over port TCP port 443 and because this is the entry point to the application which logically has other users than just Bob we're allowing TCP port 443 from any IP version for address and this means we have a security group with an inbound rule set which looks like this.

      In addition to this front-end traffic the web instance also needs to connect with the application instance and for this example let's say this is using TCP port 1337.

      Our application is that good.

      So how best to allow this communication?

      Well we could just add the IP address of the web instance into the security group of the application instance or if you wanted to allow our application to scale and change IPs then we could add the side arrangers of the subnets instead of IP addresses.

      So that's possible but it's not taking advantage of the extra functionality which security groups provide.

      What we could do is reference the web security group within the application security group.

      So this is an example of the application security group.

      Notice that it allows TCP port 1337 inbound but it references as the source a logical resource the security group.

      Now using a logical resource reference in this way means that the source reference of the A4L-web security group this actually references anything which has this security group associated with it.

      So in this example any instances which have the A4L-web security group attached to them can connect to any instances which have the A4L-web security group attached to them using TCP port 1337.

      So in essence this references this.

      So this logical reference within the application security group references the web security group and anything which has the web security group attached to it.

      Now this means we don't have to worry about IP addresses or side arrangers and it also has another benefit.

      It scales really well.

      So as additional instances are added to the application subnet and web subnet and as those instances are attached to the relevant security groups they're impacted by this logical referencing allowing anything defined within the security group to apply to any new instances automatically.

      Now this is critical to understand so when you reference a security group from another security group what you're actually doing is referencing any resources which have that security group associated with them.

      So this substantially reduces the admin overhead when you have multi-tiered applications and it also simplifies security management which means it's prone to less errors.

      Now logical references provide even more functionality.

      They allow self referencing.

      Let's take this as an example a private subnet inside AWS with an ever-changing number of application instances.

      Right now it's three but it might be three, thirty or one.

      What we can do is create a security group like this.

      This one allows incoming communications on port TCP 1337 from the web security group but it also has this rule which is a self-referential rule allowing all traffic.

      What this means is that if it's attached to all of the instances then anything with this security group attached can receive communication so all traffic from this security group and this effectively means anything that also has this security group attached to it.

      So it allows communications to occur to instances which have it attached from instances which have it attached.

      It handles any IP changes automatically which is useful in these instances within an auto scaling group which is provisioning and terminating instances based on load on the system.

      It also allows for simplified management of any intra-app communications.

      An example of this might be Microsoft the main controllers or managing application high availability within clusters.

      So this is everything I wanted to cover about security groups within AWS.

      So there's a lot of functionality and intelligence that you gain by using security groups versus network ACLs but it's important that you understand that while network ACLs do allow you to explicitly deny traffic security groups don't and so generally you would use network ACLs to explicitly block any bad actors and use security groups to allow traffic to your VPC based resources.

      You do this because security groups are capable of this logical resource referencing and that means AWS logical resources in security groups or even itself to allow this free flow of communications within a security group.

      At this point that is everything I wanted to cover in this lesson so go ahead and complete the video and when you're ready I'll look forward to you joining me in the next lesson.

    1. Welcome back and by now you should understand the difference between stateless and stateful security protection.

      In this lesson I want to talk about one security feature of AWS VPCs and a little bit more depth and that's network access control lists known as knuckles.

      Now we do have a lot to cover so let's jump in and get started.

      A network access control list we thought of as a traditional firewall available within AWS VPCs so let's look at a visual example.

      A subnet within an AWS VPC which has two EC2 instances A and B.

      The first thing to understand and this is core to how knuckles work within AWS is that they are associated with subnets.

      Every subnet has an associated network ACL and this filters data as it crosses the boundary of that subnet.

      In practice this means any data coming into the subnet is affected and data leaving the subnet is affected.

      But and this is super important to remember connections between things within that subnet such as between instance A and instance B in this example are not affected by network ACLs.

      Each network ACL contains a number of rules, two sets of rules to be precise.

      We have inbound rules and outbound rules.

      Now inbound rules only affect data entering the subnet and outbound rules affect data leaving the subnet.

      Remember from the previous lesson this isn't always matching directly to request and response.

      A request can be either inbound or outbound as can a response.

      These inbound and outbound rules are focused only on the direction of traffic not whether it's request or response.

      In fact and I'll cover this very soon knuckles are stateless which means they don't know if traffic is request or response.

      It's all about direction.

      Now rules match the destination IP or IP range, destination port or port range together with the protocol and they can explicitly allow or explicitly deny traffic.

      Remember this one network ACLs offer both explicit allows and explicit denies.

      Now rules are processed in order.

      First a network ACL determines if the inbound or outbound rules apply.

      Then it starts from the lowest rule number.

      It evaluates traffic against each individual rule until it finds a match.

      Then that traffic is either allowed or denied based on that rule and then processing stops.

      Now this is critical to understand because it means that if you have a deny rule and an allow rule which match the same traffic but if the deny rule comes first then the allow rule might never be processed.

      Lastly there's a catch all showed by the asterisk in the rule number and this is an implicit deny.

      If nothing else matches then traffic will be denied.

      So those are the basics.

      Next let's move on to some more complex elements of network ACLs.

      Now I just mentioned that network ACLs are stateless and this means that rules are required for both the request and the response part of every communication.

      You need individual rules for those so one inbound and one outbound.

      Take this example a multi-tiered application running in a VPC.

      We've got a web server in the middle and an application server on the left.

      On the right we have a user Bob using a laptop and he's accessing the website.

      So he makes a connection using HTTPS which is TCP port 443 and this is the request as you know by now and this is also going to mean that a response is required using the ephemeral port range.

      This ephemeral port is chosen at random from the available range decided by the operating system on Bob's laptop.

      Now to allow for this initial communication if we're using network ACLs then we'll need to have one associated with the web subnet and it will need rules in the inbound and outbound sections of that network ACL.

      Notice how on the inbound rule set we have rule number 110 which allows connections from anywhere and this is signified by 0.0.0.0 through this network ACL and this is allowed as long as it's using TCP port 443.

      So this is what allows the request from Bob into the web server.

      We also have on the outbound rule set rule number 120 and this allows outbound traffic to anywhere again 0.0.0.0 as long as the protocol is TCP using the port range of 1.0.2.4 to 65.5.3.5 and this is the ephemeral port range which I mentioned in the previous lesson.

      Now this is not amazingly secure but with stateless firewalls this is the only way.

      Now we also have the implicit denies and this is denoted by the rules with the star in the rule number and this means that anything which doesn't match rule 110 or 120 is denied.

      Now it's also worth mentioning while I do have rule 110 and 120 number differently the rule numbers are unique on inbound and outbound so we could have the single rule number 110 on both rule sets and that would be okay.

      It's just easier to illustrate this if I use unique rule numbers for each of the different rule sets.

      Now let's move on and increase the complexity little.

      So we have the same architecture we have Bob on the right, the web subnet in the middle and the application subnet on the left.

      You know now that because network ACLs are stateless each communication requires one request rule and one response rule.

      This becomes more complex when you have a multi-tiered architecture which operates across multiple subnets and let's step through this to illustrate why.

      Let's say the pop initiates a connection to the web server we know about this already because I just covered it.

      If we have a network ACL around the web subnet we'll need an inbound rule on the web network ACL.

      There's also going to be response traffic so this is going to use the ephemeral port range and this is going to need an outbound rule on that same web network ACL so this should make sense so far.

      But also the web server might need to communicate with the app server using some application TCP port.

      Now this is actually crossing two subnet boundaries the web at subnet boundary and the application subnet boundary so it's going to need an outbound rule on the web at subnet knuckle and also an inbound rule on the application subnet knuckle.

      Then we have the response for that as well from the app server through to the web server and this is going to be using ephemeral ports but this also crosses two subnet boundaries it leaves the application subnet which will need an outbound rule on that knuckle and it enters the web subnet which will also need an inbound rule on that network ACL and what if each of those servers need software updates it will get even more complex really quickly.

      You always have to be aware of these rule pairs the application port request and the ephemeral response for every single communication in some cases you're going to have multi-tier architecture and this might mean the communications go through different subnets.

      If you need software updates this will need more if you use network address translation or NAT you might need more rules still.

      You'll need to worry about this if you use network ACLs within a vpc for traffic to a vpc or traffic from a vpc or traffic between subnets inside that vpc.

      When a vpc is created it's created with a default network ACL and this contains inbound and outbound rule sets which have the default implicit deny but also a capsule allow and this means that the net effect is that all traffic is allowed so the default within a vpc is that knuckles have no effect they aren't used this is designed [Music] I need to be beginner friendly and reduce admin overhead.

      AWS prefer using security groups which I'll be covering soon.

      If you create your own custom network ACLs though that's a different story.

      Custom knuckles are created for a specific vpc and initially they're associated with no subnets.

      They only have one rule on both the inbound and outbound rule sets which is the default deny and the result is that if you associate this custom network ACL with any subnets all traffic will be denied so be careful with this it's radically different behavior than the default network ACL created with a vpc.

      Now this point I just want to cover some finishing key points which you need to be aware of for any real-world usage and when you're answering exam questions.

      So network access controlists remember they're known as knuckles they are stateless so they view request and response as different things so you need to add rules both for the request and for the response.

      A knuckle only affects data which is crossing the subnet boundary so communications between instances in the same subnet is not affected by a network ACL on that subnet.

      Now this can mean that if you do have data crossing between subnets then you need to make sure that each knuckle on both of those subnets has the appropriate inbound and outbound rules so you end up with a situation where one connection can in theory need two rules on each knuckle if that connection is crossing two different subnet boundaries.

      Now knuckles are able to explicitly allow traffic and explicitly deny and the deny is important because as you'll see when I talk about security groups this is a capability that you need to network ACLs.

      So network ACLs allow you to block specific IPs or specific IP ranges which are associated with bad actors so they're a really good security feature when you need to block any traffic attempting to exploit your systems.

      Now network ACLs are not aware of any logical resources they only allow you to use IPs and cyber ranges ports and protocols you cannot reference logical resources within AWS and knuckles can also not be assigned two logical resources they're only assigned to subnets within VPCs within AWS.

      Now knuckles are very often used together with security groups such as mentioned to add the capability to explicitly deny bad IPs or bad networks so generally you would use security groups to allow traffic and you use knuckles to deny traffic and I'll talk about exactly how this works in the next lesson.

      Now each subnet within a VPC has one knuckle associated with it it's either going to be the default network ACL for that VPC or a custom one which you create and associate.

      A single knuckle though can be associated with many different subnets so while a subnet can only have one network ACL one network ACL can be associated with many different subnets.

      Now this point that is everything that I wanted to cover about network ACLs for this lesson so go ahead complete the video and when you're ready I'll look forward to you joining me in the next lesson.

    1. Welcome back.

      In this lesson, I want to introduce the term "service models", specifically "cloud service models".

      If you've ever heard or seen the term something as a service or x-a-a-s, then this is generally a cloud service model, and that's what I want to cover in this lesson.

      Before I start, there are a few terms I'd like to introduce, which will make the rest of this lesson make more sense as well as being helpful throughout the course.

      If you already know these, then that's fine.

      It will just be a refresher.

      But if these are new, then it's important to make sure you understand all these concepts because they're things that underpin a lot of what makes the cloud special.

      Now, when you deploy an application anywhere, it uses what's known as an infrastructure stack.

      An infrastructure stack is a collection of things, which that application needs, all stack on to each other.

      Starting at the bottom, everything runs inside a facility, which is a building with power, with aircon, with physical security.

      Everything uses infrastructure, so storage and networking.

      An application generally requires one or more physical service.

      These servers run virtualization, which allows them to be carved up into virtual machines.

      These virtual machines run operating systems.

      They could potentially run containers.

      An example of this is Docker.

      Don't worry if you don't know what these are, I'll be covering them later in the course.

      Every application is written in a language such as Python, JavaScript, C, C++, C#, and all of these have an environment that they need to run in.

      This is called a runtime environment.

      An application needs data to work on, which it creates or consumes.

      And then at the very top is the application itself.

      Now, all of this together is an infrastructure stack or an application stack.

      If you use Netflix or Office 365 or Slack or Google or your online bank or this very training site, it has parts in each of these tiers.

      Even the application that you're using right now to watch this training is running on an operating system, which is running itself on a laptop, a PC or a tablet, which is just hardware.

      The hardware uses infrastructure, your internet connection, and this runs in facilities, so your house or a coffee shop.

      With any implementation of this stack, there are parts of the stack that you manage and there are parts of the stack which are managed by the vendor.

      So if you're working in a coffee shop, they'll have specific people to manage the building and the internet connection, and that's probably not you.

      But it's more likely that you are responsible for your laptop and the operating system running on top of it.

      Now, this is true for any system.

      Some parts you manage, some parts some people else manages.

      You don't manage any part of Netflix, for example.

      Netflix is an entity, manage everything end to end.

      But if you do work in IT, then maybe you do manage all of the IT infrastructure stack or some parts of it.

      The last term that I want to introduce is what's known as the unit of consumption.

      It's what you pay for and it's what you consume.

      It's the part of the system where from that point upwards in the infrastructure stack, you are responsible for management.

      For example, if you procure a virtual server, then your unit of consumption is the virtual machine.

      A virtual machine is just an operating system and an allocation of resources.

      So your unit of consumption is the operating system.

      In AWS, if you create a virtual machine known as an instance, then you consume the operating system.

      If you use Netflix, though, then you consume the service and that's it.

      You have no involvement in anything else.

      The unit of consumption is what makes each service model different.

      So let's take a look.

      With an on-premise system, so that's one which is running in a building that your business owns, your business has to buy all parts of the stack.

      It has to manage them all, pay for the upkeep and running costs of all of them.

      And it has to manage the staff costs and risks associated with every single part of that stack.

      Now, because it owns and controls everything, while it's expensive and it does carry some risks, it's also very flexible.

      In theory, you can create systems which are tailor-made for your business.

      Now, before cloud computing became as popular as it is now, it was possible to use something called data center hosting.

      Now, this is similar to on-premises architectures, but when you use data center hosting, you place your equipment inside a building which is owned and managed by a vendor.

      This meant that the facilities were owned and controlled by that vendor.

      You as a business consumed space in that facility.

      Your unit of consumption was a rack space.

      If you rented three racks from a data center provider, they provided the building, the security, the power, the air conditioning and the staffing to ensure the service you paid for was provided.

      All of the service models we use today are just evolutions of this type of model where more and more parts of the stack are handed off to a vendor.

      Now, the cost change, the risks that are involved change and the amount of flexibility you have changed, but it's all the same infrastructure stack just with different parts being controlled by different entities.

      So let's look at this further.

      The first cloud service model that I want to talk about is infrastructure as a service or IaaS.

      With this model, the provider manages the facilities, the storage and networking, the physical server and the virtualization and you consume the operating system.

      Remember, a virtual machine is just an operating system with a certain amount of resources assigned.

      It means that you still have to manage the operating system and anything above the operating system so any containers, the runtime, the data and your applications.

      So why use IaaS as a service model?

      With IaaS, you generally pay per second, per minute per hour fee for the virtual machine.

      You pay that fee when you use that virtual machine and you don't pay when you don't use it.

      The costs associated with managing a building, procuring and maintaining infrastructure and hardware and installing and maintaining a virtualization layer are huge and they're all managed by the vendor.

      The vendor needs to purchase things in advance, pay licenses, pay staff to keep things running and manage the risks of data loss, hardware failure and a wealth of other things.

      Using IaaS means that you can ignore all of those and let the vendor manage them.

      IaaS is one of the most popular cloud service models.

      Now, you do lose a little bit of flexibility because you can only consume the virtual machine sizes and capabilities that the provider allows, but there is a substantial cost reduction because of that.

      In AWS, a product called Elastic Compute Cloud or EC2 uses the IaaS service model.

      So in summary, IaaS is a great compromise.

      You do lose a little bit in terms of flexibility, but there are substantial costs and risk reductions.

      Okay, so let's move on.

      Another popular service model is Platform as a Service or Pass.

      Now, this service model is aimed more at developers who have an application they just want to run and not worry about any of the infrastructure.

      With Pass, your unit of consumption is the runtime of the runtime environment.

      So if you run a Python application, you pay for a Python runtime environment.

      You give the vendor some data and your application and you put it inside this runtime environment and that's it.

      You manage your application and its data and you consume the runtime environment, which effectively means that the provider manages everything else, containers, operating system, virtualization, service, infrastructure and facilities.

      Now, let's review one final service model before we finish this lesson.

      The final service model is Software as a Service or SaaS.

      And with SaaS, you consume the application.

      You have no exposure to anything else.

      You pay a monthly fee for consuming the application.

      You get it as a service.

      Now, examples of SaaS products include Netflix, Dropbox, Office 365, Flickr, even Google Mail.

      Businesses consume SaaS products because they are standard known services.

      Email is email.

      One email service is much like another.

      And so a business can save significant infrastructure costs by consuming their email service as a SaaS solution.

      They don't have much control of exactly how the email services can be configured, but there are almost no risks or additional costs associated with procuring a SaaS service.

      IaaS, SaaS and SaaS are examples of cloud service models.

      Now, there are others such as Function as a Service, known as SaaS, Container as a Service, Database as a Service or DBAAS, and there are many more.

      For this lesson, the important points to understand are that the infrastructure stack exists in every service and application that you use.

      The part of the stack is managed by you.

      The part of the stack is managed by the provider.

      And for every model, there is part of the stack which you consume, your unit of consumption.

      That's the part that you pay for and generally the part that delineates between where the vendor manages and where you manage.

      Now, again, I know this has been a fairly theory heavy lesson, but I promise you it will be invaluable as you go through the course.

      Thanks for listening.

      Go ahead, complete this video.

      And when you're ready, join me in the next.

    1. In this lesson, I want to cover theoretical topic which is really important to me personally, and something that I think is really valuable to understand.

      That is, what is multi- and hybrid Cloud, and how do they relate to private and public Cloud platforms?

      Now, why this matters is because AWS Azure and the Google Cloud Platform, they're all offering private Cloud environments which can be used in conjunction with their public Clouds.

      So to be able to pick when and where to use them effectively, you need to understand when something is multi-Cloud and when it's hybrid Cloud because these are very different things.

      So let's jump in and get started.

      In the previous lesson, I covered the formal definition of Cloud computing.

      Now, I know this was a very dry theoretical lesson, but hopefully you've come out of that understanding what a Cloud environment is.

      Now, public Cloud, simply put, is a Cloud environment that's available to the public.

      Many vendors are currently offering public Cloud platforms including AWS, Microsoft Azure, and Google Cloud.

      These are all examples of public Cloud platforms.

      They're public Cloud because they meet the five essential characteristics of Cloud computing and that they're available to the general public.

      So to be public Cloud, it needs to first classify as a Cloud environment and then it needs to be available to the general public.

      Now, you can if you have very specific needs or if you want to implement something which is highly available, you can choose to use multiple public Cloud platforms in a single system.

      Now, that's known as multi-Cloud.

      So multi-Cloud is using multiple Cloud environments and the way that you implement this can impact how successful it is.

      Now, keeping things simple, you could choose to implement a simple mirrored system.

      One part of your system could be hosted inside AWS and the other in Azure.

      This means that you've got Cloud provider level resilience.

      If one of these vendors fails, you'll know that at least part of your system will remain fully functional and running in the other.

      Now, with regards to multi-Cloud, I would personally stay away from any products or vendors who attempt to provide a so-called single management window or single pane of glass if you want to use the jargon when using multiple Cloud platforms.

      It is possible to manage multiple Cloud platforms as one single environment, but while it is possible, it abstracts away from these individual environments, relying on the lowest common feature set.

      And so you do lose a lot of what makes each vendor special and unique.

      So in this example, I could pick AWS and Azure.

      I could abstract away from that using a third-party tool.

      And when I wanted to provision a virtual machine, that tool would select which Cloud vendor to use.

      The problem with that is that it would have to assume a feature set which is available in both of them.

      So if AWS had any features that weren't available in Azure or vice versa, this third-party tool could not utilize them while staying abstracted away.

      So that's a really important thing to understand.

      Generally, when I'm thinking about multi-Cloud environments, I'm looking at it from a highly available perspective.

      So putting part of my infrastructure in one and part in another.

      It's much simpler and generally much more effective.

      Now, each of these three Cloud vendors also offers a solution which can be dedicated to your business and run from your business premises.

      This is a so-called private Cloud.

      Now, for AWS, this is called AWS Outposts.

      For Azure, it's the Azure Stack.

      And for Google, it's Anthos.

      Now, I want to make a very special point of highlighting that there is a massive difference between having on-premises infrastructure, such as VMware, Hyper-V, or Zen Server, versus having a private Cloud.

      A private Cloud still needs to meet the five essential characteristics of Cloud computing, which most traditional infrastructure platforms don't.

      So private Cloud is Cloud computing, which meets these five characteristics, but which is dedicated to you as a business.

      So with VMware, Hyper-V, or Zen Server implementation, they're not necessarily private Cloud.

      A lot of these platforms do have private Cloud-like features, but in general, the only environments that I consider true private Cloud are Outposts, the Azure Stack, and Google Anthos.

      Now, it is possible to use private Cloud in conjunction with public Cloud.

      And this is called hybrid Cloud.

      It's hybrid Cloud only if you use a private Cloud and a public Cloud, cooperating together as a single environment.

      It's not hybrid Cloud if you just utilize a public environment such as AWS together with your on-premises equipment.

      Now, to add confusion, you might hear people use the term hybrid environment.

      And in my experience, people use hybrid environment to refer to the idea of public Cloud used together with existing on-premises infrastructure.

      So I'm going to try throughout this course to have separate definitions.

      If I use the terms hybrid environment or hybrid networking, then that's different.

      That simply means connecting a public Cloud environment through to your on-premises or data-center-based traditional infrastructure.

      So there's a difference between hybrid Cloud, which is a formal definition, and then hybrid environment or hybrid networking.

      So try and separate those and understand what's meant by age.

      With true hybrid Cloud, you get to use the same tooling, the same interfaces, the same processes to interact with both the public and private components.

      So let's summarize this.

      Public Cloud means to use a single public Cloud environment such as AWS, Azure, or Google Cloud.

      Private Cloud is to use on-premises Cloud.

      Now, this is important.

      This is one of the most important distinctions to make.

      For it to be private Cloud, you need to be using an on-premises real Cloud product.

      It needs to meet those five essential characteristics.

      Multi Cloud means using more than one public Cloud.

      So an example of this might be AWS and Azure, or AWS, Azure and Google.

      They're examples of a multi Cloud deployment.

      So using multiple public Clouds in one deployment, that's a multi Cloud environment.

      And I mentioned that earlier in the lesson, that can be as simple as deploying half of your infrastructure to one public Cloud and half to the other, or using a third party tool that abstracts away from a management perspective.

      But I would not recommend any abstraction or any third party tools.

      Generally, in my experience, the best multi Cloud environments are those which use part of your infrastructure in one Cloud environment and part in the other.

      Hybrid Cloud means utilizing public and private Clouds, generally from the same vendor, together as one unified platform.

      And then lastly, and probably personally one of the most important points to me, Hybrid Cloud is not utilizing a public Cloud like AWS and connecting it to your legacy on-premises environment.

      That is a hybrid environment or a hybrid network.

      Hybrid Cloud is a very specific thing.

      And I'm stressing this because it is important now to be an effective solutions architect.

      You need to have a really good distinction between what public Cloud, private Cloud, multi Cloud, hybrid Cloud and hybrid environments are.

      Understand all of those separate definitions.

      Now that's all I wanted to cover in this lesson.

      I hope it wasn't too dry.

      I really do want to make sure that you understand all of these terms on a really foundational level because I think they're really important to be an effective solutions architect.

      So go ahead, complete this lesson and then when you're ready, I'll see you in the next.

    1. In this lesson, I want to introduce Cloud Computing.

      It's a phrase that you've most likely heard, and it's a term you probably think you understand pretty well.

      Cloud Computing is overused, but unlike most technical jargon, Cloud Computing actually has a formal definition, a set of five characteristics that a system needs to have to be considered cloud, and that's what I want to talk about over the next few minutes in this lesson.

      Understanding what makes Cloud 4 Cloud can help you understand what makes Cloud special and help you design cloud solutions.

      So let's jump in and get started.

      Now because the term Cloud is overused, if you ask 10 people what the term means, you'll likely get 10 different answers.

      What's scary is that if those 10 individuals are technical people who work with Cloud day to day, often some of those answers will be wrong.

      Because unlike you, these people haven't taken the time to fully understand the fundamentals of Cloud Computing.

      To avoid ambiguity, I take my definition of Cloud from a document created by NIST, a NIST at the National Institute of Standards and Technology, which is part of the US Department of Commerce.

      NIST creates standards documents, and one such document is named Special Publication 800-145, which I've linked in the lesson text.

      The document defines the term Cloud.

      It defines five things, five essential characteristics, which a system needs to meet in order to be cloud.

      So AWS, Azure, and Google Cloud, they all need to meet all five of these characteristics at a minimum.

      They might offer more, but these five are essential.

      Now some of these characteristics are logical, and so they may surprise you.

      So I've added a couple of things to the document, and I've added a couple of things to the document, and even though you and other business are probably sharing physical hardware, you would never know each other existed, and that's one of the benefits of Boolean.

      But on to characteristic number four, which is rapid elasticity.

      The NIST document defines this as capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly, outward, and inward, commensurate with demand.

      To the consumer, the capabilities available for provisioning often appear to be unlimited, and can be appropriated in any quantity at any time.

      Now I simplify this again into two points.

      First, capabilities can be elastically provisioned and released to scale rapidly, outward and inward with demand, and in this case, capabilities are just resources.

      And second, to the consumer, the capabilities available for provisioning often appear to be unlimited.

      Now when most people think about scaling in terms of IT systems, they see a system increasing in size based on organic growth.

      Elasticity is just an evolution of that.

      A system can start off small, and when system load increases, the system size increases.

      But, crucially, with elasticity, when system load decreases, the system can reduce in size.

      It means that the cost of a system increases as demand increases, and the system scales, and decrease as demand drops.

      Rapid elasticity is this process but automated, so the scaling can occur rapidly in real time with no human interaction.

      Cloud vendors need to offer products and features, which monitor load, and allow automated provisioning and termination as load increases and decreases.

      Now most businesses won't care about increased system costs.

      If, for example, during sale periods, their profits increase, and because the system scales, along with that increased load and increased profits, the customers are kept happy.

      Elasticity means that you don't have to, and indeed can't over provision, because over provisioning weighs money.

      It also means that you can't under provision and experience performance issues for your customers.

      It's how a company like Amazon.com or Netflix can easily handle holiday sales, or handle the load generated on the latest episode of Game of Thrones' release.

      The second part is related to that.

      A cloud environment shouldn't let you see capacity limits.

      If you need 100 virtual machines or 1000, you should be able to get access to them immediately when required.

      In the background, the provider is handling the capacity in a pooled way, but from your perspective, you should never really see any capacity limitations.

      Now this is, in my opinion, the most important benefit of cloud, systems which scale in size in response to load.

      So this is a really important one to make sure that a potential cloud environment offers in order to make sure that it is actually cloud.

      Okay, let's move on to the final characteristic, and that's measured service.

      Now this document defines this as cloud systems automatically control and optimize resource use by leveraging and metering capability at some level of abstraction appropriate to the type of service.

      And it says that resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the service.

      Now my simplified version of this is that resource usage can be monitored, controlled, reported, and built.

      Traditional and non-cloud infrastructure work to using capex.

      You pay for service and hardware in advance.

      In the beginning, you had more capacity than you needed, so money was wasted.

      Your demand grew over time, and eventually you purchased more service to cope with the demand.

      If you did that too slowly, you had performance issues or failures.

      With a true cloud environment, it offers on-demand billing.

      Your usage is monitored on a constant basis.

      You pay for that usage.

      This might be a certain amount per second, a minute, an hour, or per day usage of a certain service, for example virtual machines.

      Or it could be a certain cost for every gigabyte you store on a storage service for a given month.

      You generally pay nothing in advance if it truly is a cloud platform.

      If you consume a virtual server for a month, but then for 30 minutes at that month you use 100 virtual servers, then you should pay a small amount for the month and a much larger amount just for that 30 minutes.

      Legacy vendors will generally want to feed or buy or lease a server.

      If this is the case, they aren't cloud, and they probably don't support some of the massively flexible architectures the cloud allows you to build.

      With that being said, that is everything I wanted to cover, so go ahead, complete this video, and when you're ready, I'll join you in the next.

    1. Welcome back.

      In a previous video I talked about YAML, which is a method of storing and passing data which is human readable.

      In this video I want to cover JSON, which is the JavaScript object notation.

      Let's jump in and take a quick look and note that many of the topics which I covered in the YAML video also apply here.

      With conveying the same information, the format to do so is just different.

      So unfortunately we have another mouthful of definition incoming.

      So JSON, or the JavaScript object notation, is a lightweight data interchange format.

      It's easy for humans to read and write and it's easy for machines to pass and generate.

      That's what it says, but there are a few differences that you should be aware of before we move on.

      JSON doesn't really care about indentation because everything is enclosed in something, so braces or brackets.

      Because of this it can be much more forgiving regarding spacing and positioning.

      And secondly because of that, JSON can appear initially harder to read.

      But over time I've come to appreciate the way that JSON lays out the structure of its documents.

      Now there are two main elements that you need to understand if you want to be competent with JSON.

      First, an object or a JSON object.

      And this is an unordered set of key value pairs enclosed by curly brackets.

      Now from when you watched the YAML video, you should recognise this as a dictionary.

      It's the same thing, but in JSON it's called an object.

      The second main element in JSON is an array which is an ordered collection of values separated by commas and enclosed in square brackets.

      Now from the YAML video you might recognise this as a list.

      It's again the same thing, only in JSON it's called an array.

      Now in both cases, arrays which are lists of values or objects which are collections of key value pairs, the value can be a string, an object, a number, an array, boolean, true or false, or finally null.

      Now with these two high level constructs in mind, let's move on.

      So this is an example of a simple JSON document.

      Notice how even at the top level there are these curly brackets.

      This shows that at the top level a JSON document is simply a JSON object, a collection of key value pairs separated by a colon.

      In this example we have three keys.

      We have cats, colours and finally, num of eyes.

      And each key has a corresponding value in this example which is an array.

      The top level key value pair has a value containing an array of cat names.

      The middle has a value which is an array of the colour of the cats.

      And then the last key value pair has a value which is a list of the number of eyes which each cat has.

      Now JSON documents aren't limited to just arrays.

      They can be much more complicated like this example.

      Now this is a JSON document and every JSON document starts with a top level object, which is an unordered list of key value pairs surrounded by curly brackets.

      This object has four key value pairs.

      The keys are ruffle, truffles, penny and winky.

      The value of each key is a JSON object, a collection of key value pairs.

      So JSON objects can be nested within JSON objects, arrays can be ordered lists of JSON objects, which themselves can contain JSON objects.

      And again, this lets you create complex structures which can be used by applications to pass or store data and configuration.

      Now I'll admit it, I'm actually a fan of JSON.

      I think it's actually easier to write and read than YAML.

      Many people will disagree and that's fine.

      With that being said though, that's everything that I wanted to cover in this video.

      So go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome to this video which will be a fairly high level introduction to YAML.

      Now YAML stands for YAML 8 Markup Language and for any key observers that's a recursive acronym.

      Now I want this video to be brief but I think it's important that you understand YAML's structure.

      So let's jump in and get started.

      YAML is a language which is human readable and designed for data serialization.

      Now that's a mouthful but put simply it's a language for defining data or configuration which is designed to be human readable.

      At a high level a YAML document is an unordered collection of key value pairs separated by a colon.

      It's important that you understand this lack of order.

      At this top level there is no requirement to order things in a certain way.

      Although there may be conventions and standards none of that is imposed by YAML.

      An example key value pair might be the key being cat1 and the value being raffle.

      One of my cats in this example both the key and the value are just normal strings.

      We could further populate our YAML file with a key of cat2 and a value of truffles and other cat of mine.

      Or a key of cat3 and a value of penny and a key of cat4 and a value of winkey.

      These are all strings.

      Now YAML supports other types numbers such as one and two, floating point values such as 1.337, boolean so true or false and even null which represents nothing.

      Now YAML also supports other types and one of those are lists known as arrays or other names depending on what if any programming languages that you're used to.

      A list is essentially an ordered set of values and in YAML we can represent a list by having a key let's say Adrian's cats.

      And then as a value we might have something that looks like this, a comma separated set of values inside swear brackets.

      Now this is known as inline format where the list is placed where you expect the value to be after the key and the colon.

      Now the same list can also be represented like this where you have the key and then a colon and then you go to a new line and each item in the list is represented by hyphen and then the value.

      Now notice how for some of the values are actually enclosed in speech marks or quotation marks and so on.

      This is optional.

      All of these are valid.

      Often though it's safe for you to enclose things as it allows you to be more precise and it avoids confusion.

      Now in YAML indentation really matters.

      Indentation is always done using spaces and the same level of indentation means that the things are within the same structure.

      So we know that because all of these list items are indented by the same amount they're all part of the same list.

      We know they're a list because of the hyphens.

      So same indent always using hyphens means that they're all part of the same list, same structure.

      Now these two styles are two methods for expressing the same thing.

      A key called Adrian's cats whose value is a list.

      This is the same structure.

      It represents the same data.

      Now there's one final thing which I want to cover with YAML and that's a dictionary.

      A dictionary is just a data structure.

      It's a collection of key value pairs which are unordered.

      A YAML template has a top level dictionary.

      It's a collection of key value pairs.

      So let's look at an example.

      Now this looks much more complicated but it's not if you just follow it through from the start.

      So we start with a key value pair.

      Adrian's cats at the top.

      So the key is Adrian's cats and the value is a list.

      And we can tell that it's a list because of the hyphens which are the same level of indentation.

      But, and this is important, notice how for each list item we don't just have the hyphen and a value.

      Instead we have the hyphen and for each one we have a collection of key value pairs.

      So for the final list item at the bottom we have a dictionary containing a number of key value pairs.

      The first has a key of name with a value of winky.

      The second a key color with a value of white.

      And then for this final list item a key, num of eyes and a value of one.

      And each item in this list, each value is a dictionary.

      A collection of one or more key value pairs.

      So values can be strings, numbers, floats, booleans, lists or dictionaries or a combination of any of them.

      Note how the color key value pair in the top list item, so the raffle dictionary at the top, its value is a list.

      So this structure that's on screen now, we have Adrian's cats which are a value, has a list.

      Each value in the list is a dictionary.

      Each dictionary contains a name, key, with a value, a color key, with a value.

      And then the third item in the list also has a num of eyes key and a value.

      Now using YAML key value pairs, lists and dictionaries allows you to build complex data structures in a way which once you have practice is very human readable.

      In this case, it's a database of somebody's cats.

      Now YAML can be read into an application or written out by an application.

      And YAML is commonly used for the storage and passing of configuration.

      For now thanks for watching, go ahead, complete the video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome to this video where I'm going to step through two concepts which I think should be mandatory knowledge for any solutions architect.

      And for anyone else working in IT, these are also really useful.

      First we have recovery point objective known as RPO and recovery time objective known as RTO.

      Generally if you're a solutions architect helping a client, they will give you their required values for both of these.

      In some cases you might need to work with key stakeholders within the business to determine appropriate values.

      In either case if you get them wrong it can have a massive negative consequence to a business.

      Let's jump in and get started.

      I'm going to start by stepping through recovery point objective or RPO.

      Recovery point objective or RPO is something that's generally expressed in minutes or hours.

      And I'll illustrate this, let's say a given 24 hour period.

      It starts on the left and midday, moves through midnight in the middle and finishes at 12 midday on the following day on the right.

      I will need to consider an animal rescue business who have animals arriving to be fostered 24/7/365.

      They have intake, vet exams and data restored within on-premises systems which need to be referred to constantly throughout the day.

      At a certain point in time let's say 2am we have a server failure.

      And for this example let's assume this is a single server which stores all the data for the organization and they have no redundancy.

      Now this is a terrible situation but it's all too common for cash-strapped charities.

      So remember, donate to your local animal rescue centre.

      RPO is defined as the maximum amount of data and this is generally expressed in time that can be lost during a disaster recovery situation before that loss will exceed what the organization can tolerate.

      If an organization tells you that they have an RPO of 6 hours, it means the organization cannot tolerate more than 6 hours of data loss when recovering from a disaster like this server failure.

      Now different organizations will have different RPO values.

      Banks logically will be able to tolerate although some know data loss because they deal with customer money.

      Whereas an online store might be able to tolerate some data loss as they can in theory recreate orders in other ways.

      Understanding how data can be lost during disaster recovery scenarios is key to understanding how to implement a given RPO requirement.

      Let's consider this scenario that every 6 hours starting at 3pm on day 1 the business takes a full backup of the server which has failed.

      So normally we have a backup at 3pm, one at 9pm, one at 3am and one at 9am.

      So 4 backups every 24 hour period split by 6 hours.

      In order to recover data from the failed server we need to restore a backup.

      Ideally assuming that we have no failures it will be from the most recent backup.

      Now successful backups are known as recovery points.

      In the case of full backups each successful backup is one recovery point.

      If you use full backups and incremental backups it's possible that to restore a single incremental backup, i.e. to use that one recovery point you'll need the most recent full backup and every incremental backup between that full and the most recent okay incremental backup.

      So it's possible that a recovery point will need more than one backup.

      With this scenario so the server failure at 2am the data loss will be the time between 2am and the most recent recovery point.

      In this case 9pm at the previous day.

      So this represents 5 hours of lost data.

      If the failure occurred right after the 9pm backup had finished we'd have almost no data loss.

      If the failure occurred one hour later at 3am we would have 6 hours of data loss.

      Now the maximum loss of data for this type of scenario is the time between 2 successful backups.

      In our case because backups occur every 6 hours then data loss could be a minimum of 0 if the server failure occurred right after the first backup finished or a maximum of 6 hours if the server failure occurred right before the next scheduled backup.

      So when an organisation informs you that they have a requirement for an RPO of 6 hours they're telling you that they can only tolerate a maximum of 6 hours of data loss in a disaster scenario.

      And as a general rule this means that you need to make sure that backups occur as often or more often than the RPO value provided by the organisation.

      An RPO of 6 hours means at minimum a backup every 6 hours but to cope with random backup failure generally you'll want to make sure backups occur more frequently than required.

      So in this example maybe once every 3 hours or maybe even once an hour.

      Lower RPOs generally require more frequent backups which historically has resulted in higher cost for backup systems both in terms of media which also licensing, management overhead and other associated processes.

      So RPO is a value which is generally given to you by an organisation or you might have to work with an organisation to identify an appropriate value and it states how much maximum loss of data in time the business can tolerate.

      Different businesses will have different RPOs and sometimes even different RPOs for different systems within a single organisation.

      A bank might have super low RPOs for its financial systems but it might tolerate a much higher one for its website.

      If data changes less frequently the system is less important then higher RPO values are easier to tolerate for a business.

      Now let's move on and cover recovery time objective or RTO.

      To explain RTO we're going to use the same example of a 24 hour period starting at midday on one day moving through that day with midnight in the middle and then moving to midday the following day on the right.

      For this example though I've moved the server failure to 10pm on day one and the most recent backup was 9pm so one hour before the failure.

      As you know now this means assuming the backup is working i.e. it's a valid recovery point that the data loss will be one hour, the time between the 9pm backup and the 10pm server failure.

      RTO or recovery time objective simply put is the maximum tolerable length of time that a system can be down after a failure or disaster occurs.

      Now once again just as with RPO this value is something that a business will give you as a directive or alternatively it's something that you'll work with a business on to determine a suitable value.

      Also just as with RPOs different businesses will have different RPOs.

      A bank will have a much lower RPO for its banking systems than a cafe for its website and an organisation will generally have different RPOs for its different systems.

      Critical systems will have lower RPOs and less important systems can potentially have higher RPOs.

      Now looking at the RPO definition in a different way, if the animal rescue business had an RPO of 13 hours it would mean that for a server failure which occurred at 10pm the IT operations team would have as a maximum until 11am the following day to fully restore the system to an operational state.

      Now something which is really important that I need you to understand and this isn't always obvious especially if you haven't worked on or with a service desk before.

      Recovery time of the system begins at the moment of the failure that's when the clock starts ticking and it ends not when the issue is fixed but when the system is handed back to the business in a fully tested state.

      So in this example the clock starts at 10pm and to meet an RPO of 13 hours the server needs to be running again fully working by 11am the following day.

      So you might ask why I'm stressing this point?

      Well RPO isn't just about technical things, the biggest impacts on RPOs are things which as a technical person you might not always identify.

      I want to step through some considerations which might impact the ability of this animal rescue's operation team to meet this 13 hour RPO directive.

      The first thing which might not be immediately obvious is that to recover a system you need to know that that system has failed.

      If the failure occurs at 10pm while the recovery time starts at that point the ability to recover only really starts when you're made aware that the system has failed.

      So how long till the operations team know that there is an issue?

      Is there monitoring in place on this service?

      Is it reliable because too many false positives will have people ignoring outage notifications?

      How will the monitoring system notify staff?

      Will it wake staff who are sleeping?

      Will it be the correct staff, staff who are empowered to begin a recovery process?

      This is the real starting point to any recovery process and it often adds lag onto the start of the process.

      Even with major internet applications that I use it's not uncommon for outages to occur and then take a further 15 to 30 minutes before the vendor is actively aware and investigating a fault.

      Now don't underestimate the importance of effective monitoring and notification systems.

      Beyond that make sure that you've planned and configured these processes in advance.

      Waking up a junior operations person who has no ability to make decisions or no ability to wake up senior staff members is useless in this scenario.

      Best case this part of the process takes some time so make sure that it's built into your planning.

      Now let's move on and assume that we do have somebody in the ops team who can begin the process.

      Well step number two is going to be to investigate the issue.

      It might be something which is fixable quickly or it might be that a server is literally on fire.

      Somebody needs to take the time to make the final decision to perform a restore if required and again this will take some time.

      Moving on if we assume that we are going to do a restore we need to focus on the backup system.

      What type of backups do we have?

      Some take longer to restore versus others.

      If it's a tape backup system where are the tapes?

      Where is the tape drive or the loader?

      Who needs to restore it?

      Do they need to be in a specific physical location?

      How does the restore happen?

      Is there a documented process?

      And is the person or one of the people who can perform the restore available and awake?

      All of these are critical in your ability to begin the restore process and they all take time.

      Now this type of disaster recovery scenario if you don't have a documented and tested process this can also be really stressful.

      And a stressful situation late at night without your team around you this is when mistakes happen.

      But let's assume that we have a working backup system.

      We know where the backup media is and we have that and somebody who can operate the restore.

      The next step is where we're restoring to.

      The server we had has had a major failure or might literally be on fire.

      Do you think about this in advance?

      These choices so what are we restoring on?

      Do we have a spare?

      Do we need to order another server?

      Are we using physical or virtual servers?

      Or are we even forced to use a secondary disaster recovery site because not only is the server on fire but also is the server room.

      Many people miss these elements when thinking about RTO but these are the things which really matter.

      A badly documented process to restore servers or a slow notification system might add additional hours.

      But having to order new server hardware could add days to a recovery time.

      And finally for those wells that the restore has completed the operations team thinks the service is back up and running.

      Then there needs to be time allocated for business testing, user testing and final handover.

      This isn't quick and has to be done before you consider recovery to be complete.

      This entire process end to end is what recovery is.

      And so if the business has a 13 hour RTO you need to make sure that all of this process in its entirety fits into that 13 hours.

      So that's what RTO is, a value given to you by a business or something that you help a business identify.

      It's the maximum tolerable time to recover a system in the event that a disaster occurs.

      It's the end to end process so this includes fault identification, restoration and final testing and handover.

      So it's really important that when you're planning the recovery process for a system and you're given an RTO value by the business you're sure that you have time to perform all of these individual steps.

      Now let's quickly summarize what we've learned before we finish this video.

      So RPO is how much data the maximum data expressed in time at a business can lose.

      So this is amount of data expressed in time beyond which it's not tolerable to the business.

      So worst case this is the time between successful backups.

      In general to implement more and more demanding RPO directives you need more frequent backups.

      This means more cost but it does result in a lower RPO.

      So when you see RPO think maximum data loss.

      RTO or recovery time objective is a directive from the business which is a maximum restore time that that business can tolerate.

      And this is end to end from identification through to final testing and handover.

      So this can be reduced by effective planning, monitoring, notification, formal processes, spare hardware, training and more efficient systems such as virtual machines or AWS.

      So RTO is the maximum time from when a failure occurs through to when the business will need that system back up and running in an operational state.

      And by thinking about this in advance you can make your recovery process more efficient and meet more demanding RTOs from the business.

      Now different businesses and different systems within the businesses will have different RPO and RTO values.

      Generally the more critical a system is the lower and thus more demanding the RPO and RTO values will be.

      And non-critical systems of business is usually more willing to tolerate higher and so less demanding RPO and RTO values.

      Because generally what you're looking for is a gold lock point where you're as close to the true business requirements as possible.

      Now as a solutions architect it's often the case that the business isn't aware of appropriate RTO and RTO values.

      And so one of the core duties when designing new system implementations is to work with the business and understand which systems are critical and which can tolerate more data loss or recovery outages.

      And by appropriately designing systems to match the true business requirements you can deliver a system which meets those requirements in a cost effective way.

      Now at this point that's everything I want to cover about RPO and RTO at a high level.

      If you're doing one of my ADL West courses as you're going through the course consider how you think the products and services being discussed would affect the RPO and RTO's of systems designed utilizing those products.

      And if appropriate I'll be discussing exactly how features of those products can influence RPO and RTO values.

      At this point though that's everything I wanted to cover in this video.

      Thanks for watching.

      Go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome back.

      In this fundamentals video, I want to briefly talk about Kubernetes, which is an open source container orchestration system.

      You use it to automate the deployment, scaling and management of containerized applications.

      At a super high level, Kubernetes lets you run containers in a reliable and scalable way, making a vision fuse of resources, and lets you expose your containerized applications to the outside world or your business.

      It's like Docker, only with robots automated and super intelligence for all of the thinking.

      Now, Kubernetes is a cloud agnostic product, so you can use it on premises and within many public cloud platforms.

      Now, I want to keep this video to a super high level architectural overview, but that's still a lot to cover.

      So let's jump in and get started.

      Let's quickly step through the architecture of the Kubernetes cluster.

      A cluster in Kubernetes is a highly available cluster of compute resources, and these are organized to work as one unit.

      The cluster starts with a cluster control plane, which is the part which manages the cluster.

      It performs scheduling, application management, scaling and deployment, and much more.

      Compute within a Kubernetes cluster is provided via nodes, and these are virtual or physical servers, which function as a worker within the cluster.

      These are the things which actually run your containerized applications.

      Running on each of the nodes is software, and at minimum, this is container D or another container runtime, which is the software used to handle your container operations.

      And next, we have KubeLit, which is an agent to interact with the cluster control plane.

      And on each of the nodes communicates with the cluster control plane using Kubernetes API.

      Now, this is the top level functionality of the Kubernetes cluster.

      The control plane orchestrates containerized applications which run on nodes.

      But now let's explore the architecture of control planes and nodes in a little bit more detail.

      On this diagram, I've zoomed in a little.

      We have the control plane at the top and a single cluster node at the bottom, complete with the minimum Docker and KubeLit software running for control plane communications.

      Now, on to step through the main components which might run within the control plane and on the cluster nodes.

      Keep in mind, this is a fundamental level video.

      It's not meant to be exhaustive.

      Kubernetes is a complex topic, so I'm just covering the parts that you need to understand to get started.

      Now, the cluster will also likely have many more nodes.

      It's rare that you only have one node unless this is a testing environment.

      Now, first, I want to talk about pods and pods at the smallest unit of computing within Kubernetes.

      You can have pods which have multiple containers and provide shared storage and networking for those pods.

      But it's very common to see a one-container, one-pod architecture, which as the name suggests, means each pod contains only one container.

      Now, when you think about Kubernetes, don't think about containers.

      Think about pods.

      You're going to be working with pods and you're going to be managing pods.

      The pods handle the containers within them.

      Architecturally, you would generally only run multiple containers in a pod when those containers are tightly coupled and require close proximity and rely on each other in a very tightly coupled way.

      Additionally, although you'll be exposed to pods, you'll rarely manage them directly.

      Pods are non-permanent things.

      In order to get the maximum value from Kubernetes, you need to view pods as temporary things which are created, do a job, and are then disposed of.

      Pods can be deleted when finished, evicted for lack of resources, or the node itself fails.

      They aren't permanent and aren't designed to be viewed as highly available entities.

      There are other things linked to pods which provide more permanence, but more on that elsewhere.

      So now let's talk about what runs on the control plane.

      Firstly, I've already mentioned this one, the API, known formally as Q-API server.

      This is the front end for the control plane.

      It's what everything generally interacts with to communicate with the control plane, and it can be scaled horizontally for performance and to ensure high availability.

      Next, we have ETCD, and this provides a highly available key value store.

      So a simple database running within the cluster, which acts as the main backing store for data for the cluster.

      Another important control plane component is Q-scheduler, and this is responsible for constantly checking for any pods within the cluster which you don't have a node assigned.

      And then it assigns a node to that pod based on resource requirements, deadlines, affinity, or anti-affinity, data locality needs, and any other constraints.

      Remember, nodes are the things which provide the raw compute and other resources to the cluster, and it's this component which makes sure the nodes get utilized effectively.

      Next, we have an optional component, the Cloud Controller Manager, and this is what allows Kubernetes to integrate with any cloud providers.

      It's common that Kubernetes runs on top of other cloud platforms such as AWS, Azure, or GCP, and it's this component which allows the control plane to closely interact with those platforms.

      Now, it is entirely optional, and if you run a small Kubernetes deployment at home, you probably won't be using this component.

      Now, lastly, in the control plane is the Q-Controller Manager, and this is actually a collection of processors.

      We've got the node controller, which is responsible for monitoring and responding to any node outages, the job controller, which is responsible for running pods in order to execute jobs, the endpoint controller, which populates endpoints in the cluster, more on this in a second, but this is something that links services to pods.

      Again, I'll be covering this very shortly.

      And then the service account and token controller, which is responsible for account and API token creation.

      Now, again, I haven't spoken about services or endpoints yet, just stick with me.

      I will in a second.

      Now, lastly, on every node is something called K-Proxy, known as Cube Proxy, and this runs on every node and coordinates networking with the cluster control plane.

      It helps implement services and configs rules allowing communications with pods from inside or outside of the cluster.

      You might have a Kubernetes cluster, but you're going to want some level of communication with the outside world, and that's what Cube Proxy provides.

      Now, that's the architecture of the cluster and nodes in a little bit more detail, but I want to finish this introduction video with a few summary points of the terms that you're going to come across.

      So, let's talk about the key components.

      So, we start with the cluster, and conceptually, this is a deployment of Kubernetes.

      It provides management orchestration, healing, and service access.

      Within a cluster, we've got the nodes which provide the actual compute resources, and pods run on these nodes.

      A pod is one or more containers, and it's the smallest admin unit within Kubernetes, and often, as I mentioned previously, you're going to see the one container, one pod architecture.

      Simply put, it's cleaner.

      Now, a pod is not a permanent thing, it's not long-lived.

      The cluster can and does replace them as required.

      Services provide an abstraction from pods, so the service is typically what you will understand as an application.

      An application can be containerized across many pods, but the service is the consistent thing, the abstraction.

      Service is what you interact with if you access a containerized application.

      Now, we've also got a job, and a job is an ad hoc thing inside the cluster.

      Think of it as the name suggests, as a job.

      A job creates one or more pods, runs until it completes, retries if required, and then finishes.

      Now, jobs might be used as back-end isolated pieces of work within a cluster.

      Now, something new that I haven't covered yet, and that's Ingress.

      Ingress is how something external to the cluster can access a service.

      So, you have external users, they come into an Ingress, that's routed through the cluster to a service, the service points at one or more pods, which provides the actual application.

      So, Ingress is something that you will have exposure to when you start working with Kubernetes.

      And next is an Ingress controller, and that's a piece of software which actually arranges for the underlying hardware to allow Ingress.

      For example, there is an AWS load balancer, Ingress controller, which uses application and network load balancers to allow the Ingress.

      But there are also other controllers such as Nginx and others for various cloud platforms.

      Now, finally, and this one is really important, generally it's best to architect things within Kubernetes to be stateless from a pod perspective.

      Remember, pods are temporary.

      If your application has any form of long-running state, then you need a way to store that state somewhere.

      Now, state can be session data, but also data in the more traditional sense.

      Any storage in Kubernetes by default is ephemeral, provided locally by a node, and thus, if a pod moves between nodes, then that storage is lost.

      Conceptually, think of this like instant store volumes running on AWS EC2.

      Now, you can configure persistent storage known as persistent volumes or PVs, and these are volumes whose lifecycle lives beyond any one single pod, which is using them.

      And this is how you would provision normal long-running storage to your containerized applications.

      Now, the details of this are a little bit beyond this introduction level video, but I wanted you to be aware of this functionality.

      OK, so that's a high-level introduction to Kubernetes.

      It's a pretty broad and complex product, but it's super powerful when you know how to use it.

      This video only scratches the surface.

      If you're watching this as part of my AWS courses, then I'm going to have follow-up videos which step through how AWS implements Kubernetes with their EKS service.

      If you're taking any of the more technically deep AWS courses, then maybe other deep-dive videos into specific areas that you need to be aware of.

      So there may be additional videos covering individual topics at a much deeper level.

      If there are no additional videos, then don't worry, because that's everything that you need to be aware of.

      Thanks for watching this video.

      Go ahead and complete the video, and when you're ready, I look forward to you joining me in the next.

    1. Welcome back in this video I want to talk about the DNS signing ceremony.

      If you're imagining confetti and champagne right now, it's the opposite of that kind of ceremony.

      This ceremony is all about controlling the keys to the internet, more specifically the trust anchor of the DNS system.

      It's one of the most important meetings which occur in the technical space.

      Pretty much everything that you use on the internet is enabled by the technical act which occurs within the ceremony.

      Now before I cover what the ceremony is, we need to understand why anything like this is needed.

      Trust within a DNS zone is normally provided via the parent zone of that zone.

      The parent zone has a DS record which is a hash of the public key signing key of the child zone, and that's how the trust chain is created.

      In the case of the root zone, there is no parent zone and this means there's nothing to provide that trust.

      And so a more rigorous process is required, something which is secure enough that the output can be absolutely trusted by every DNSSEC resolver and client.

      And we refer to this concept as a trust anchor.

      Locked away within two secure locations, one in California and another in Virginia, is what amounts to the keys of the internet.

      The private DNS root key signing key known as a KSK.

      Now it's impossible to overstate how important this set of keys is to the internet.

      They rarely change and the trust in them is hard coded into all DNSSEC clients.

      With them you can define what's valid on the DNSSEC root zone.

      Because of this, every child's top level domain, every child's zone, inside those and every DNS record, these are locked away, protected and never exposed.

      And they use redundant hardware security modules also redundant across physical locations.

      Now I'll detail this more in a second.

      With access to the private keys is controlled via the fact that HSMs are used.

      You can only use the HSMs in tightly controlled ways.

      The keys never leave those HSMs, those HSMs never leave those locations.

      And you can only use them within those locations if you have the right group of people.

      And people can only get into those locations after going through a rigorous multi-stage ID process.

      Now why this is important is because we all know a public part of this key.

      It's part of the DNS key record set within the root zone along with the public zone signing key.

      To recite this, every DNSSEC client and resolver on the planet explicitly trusts this key, this key signing key.

      And if we have this public root zone key signing key, we can verify anything signed by the private key, the one that's locked away within the hardware security modules.

      Because the security of the private root key signing key is so tight, it's not practical to use constantly.

      And so there's another key pair which controls the security of the DNS root zone.

      This is known as the root zone ZSK or zone signing key.

      The whole function of this massively controlled ceremony is to take the root zone ZSK, take this into the ceremony, sign it with the private root zone KSK, within these hugely tightly controlled conditions and then produce as an output the root zone RRSIG DNS key.

      This single record is why DNS from this level down through the top level domains and into the domain zones, this is why it's all trusted because the root zone is trusted via this signing process.

      Now talking through the detail of the signing ceremony would take too long.

      And so I've included a link attached to this video which gives a detailed overview of the ceremonial process.

      These ceremonies are public and recorded and links detailing all of this process is included attached to this video.

      What I want to do now is to talk in a little bit more detail about why this process is so secure.

      The signing ceremony itself takes place, as I mentioned previously, in one of two secure locations.

      Now there are a few key sets of people involved.

      I'll not show them all on screen, but we have the ceremony administrator, an internal witness, the credential safe controller, the hardware safe controller, and then crypto officer 1, crypto officer 2 and crypto officer 3.

      Now there are a total of 14 of these crypto officers.

      Seven of them are affiliated with each of the locations and at least three are required to attend for the process to work.

      Logistically, dates and times of it 24 to 5 are available to ensure some level of resilience.

      The most important part of the whole process hardware wise is the hardware security module or HSM, which is the hardware which contains the root zone private key signing key.

      This device is protected and can only be interacted with via the ceremony laptop which is connected to the HSM over ethernet and this is only operated by the ceremony administrator.

      The laptop has no battery and no storage.

      It's designed to be stateless and only used to perform the ceremony and not store any data afterwards.

      Now the HSM device can only be used when crypto officers use their allocated cards.

      What's being signed are the public key signing key and zone signing keys.

      We'll actually a pack of them to allow for rotation between this ceremony date and the next.

      The HSM device via the ceremony laptop then outputs the signatures for these keys which become the DNS key, RRC records for the DNS root zone.

      Now again this process happens every three months.

      It's generally broadcast, notes of the process are available and it's publicly audited.

      Now this is a summary of the process.

      The level of security procedure which goes into ensuring that groups of human participants can't collude and corrupt the process is extreme.

      I've included additional links attached to this video which provide more detail if you're interested.

      For this video I just want you to have an understanding of why the ceremony is so important.

      So during the ceremony we take the root key signing keys which everything trusts but which are too important to be used day to day.

      And we use those to sign root zone signing keys which can be used on a more operational basis and these can be used to sign individual resource record sets within the root zone.

      And it's that public and private zone signing key pair which is then used to create the chain of trust which allows trust in top level domains.

      And then those top level domains can pass that trust on to domains and then in domains that trust can be passed to individual resource record sets.

      And this all has to happen because we have nothing above the root key signing keys in the DNS.

      There are trust anchor.

      Nothing makes us trust them other than the trust itself.

      And the ceremony ensures that it's almost impossible to corrupt that trust.

      At this point that is everything I wanted to cover in this video.

      So go ahead and complete the video and when you're ready I look forward to you joining me in the next.

    1. Welcome back.

      In this video I want to help you understand how the chain of trust works within DNSSEC.

      Now we've got a lot to cover, so let's jump in and get started straight away.

      I want to first talk about DS records known as delegated sign records, because this is how DNSSEC creates a chain of trust between parent and child.

      On the right we have the ICANN.org zone, the one that we finished with in the previous video.

      It contains the public key signing key for that zone and some other DNS stuff.

      Now I'm not evaluating the other DNS stuff, but because of the hierarchical architecture of DNS we don't have to think about it at this level.

      All that matters when we move up a level are the ICANN.org name servers for normal DNS and the public key signing key for DNSSEC.

      On the left of the screen we have the parent zone for ICANN.org, which is the org top level domain zone.

      This is one level up in the DNS hierarchy.

      In this part of the video I want to focus on how the .org zone delegates to the ICANN.org zone for ICANN.org, and how DNSSEC delegates trust via the trust chain to this child zone.

      Now because DNSSEC builds on top of DNS we start with the delegation for DNS.

      This is in the form of name server resource records, which point at the name servers hosting the ICANN.org zone.

      This just integrates the ICANN.org domain into the DNS infrastructure.

      It doesn't provide any of the verification which we're looking for from DNSSEC.

      Now there's a matching RRC for this RRC set which verifies these records, but for the parent to child zone trust we need more.

      The .org zone needs to explicitly state that it trusts the ICANN.org child zone, and this is done using a delegated signer record set.

      And what this does is store a hash of the child domain's public key signing key in this record in the parent zone.

      Since the hash is one way and since they're unique, adding this record shows that the parent zone trusts the child's key signing key for that child zone.

      So at this point the .org zone is confirming its trust of the ICANN.org KSK via this record set.

      And like everything else relating to DNSSEC we need to provide a mechanism to validate this DS record set.

      This means a matching RRC which is a digital signature of the DS RRC, made using the .org zone's private zone signing key.

      At this point all of the records can be validated within .org and we can confirm the trust of ICANN.org.

      Now we just need to add the records which store the .org zone's public keys.

      So this means a DNS key record set containing the .org public zone signing and key signing keys.

      And these are used to validate RRC's in the zone.

      And this DNS key RRC needs a matching RRC which is created by signing the DNS key RRC with the zone's key signing key.

      At this point if you trust the .org zone's public key signing key then you trust everything in the .org zone including the trust delegation to the ICANN.org child zone.

      And how do we trust this in the same way that the child ICANN.org zone was trusted by the .org zone?

      Trust is given from the parent zone which in this case is the root zone.

      This is the process how a parent zone trusts a child via the DS record set which contains a hash of the child zone's key signing key.

      Now if we zoom out a little bit I want to talk about the DNSSEC validation flow.

      So how we build this chain of trust between the different levels of the DNS hierarchy.

      So the trust between levels from child to parent is created via public key signing keys and the DS record sets in the parent zone.

      So the ICANN.org zone it has a public key signing key.

      The .org zone validates this by having a DS record and storing a hash of that key signing key in that record set.

      The .org zone has its own key signing key.

      And the .org zone is trusted because a hash of this key is stored in the corresponding DS record in the root zone.

      At the DNS root zone level we hit a problem because there's no parent zone to provide this trust.

      The root zone has a public key signing key and this and the corresponding private root key signing key.

      These are explicitly trusted.

      I'll talk about how this explicit trust works and what happens in the video dedicated to the key signing ceremony.

      But at this stage just take it as fact that the root key signing keys are trusted explicitly.

      They're known as a trust anchor and every DNSSEC capable client or resolver by default trusts these keys.

      Because there's no parent zone to the root zone this has to be a trust anchor.

      Something that just is.

      So because we trust the root key signing keys it means that a DNSSEC capable resolver can follow this chain from root through to the DNS record set.

      So the root zone key signing key has signed the root zone zone signing key.

      This is signed the .org DS record in the root zone.

      So this can now be cryptographically validated.

      This record matches the public key signing key in the org zone.

      And so this too can be cryptographically validated.

      Inside the org zone the key signing key has signed the zone signing key and this has signed record sets including the DS record set for ICANN.org inside the .org zone.

      And so this also can be validated.

      This contains a hash of the public key signing key for the ICANN.org zone.

      And so lastly this too can also be cryptographically validated.

      And this is how you build a chain of trust from root through to record sets.

      We start with the trust anchor that's created during the key signing ceremony that I'm going to be talking about in a separate video.

      And then at every level of the DNS hierarchy by using DS record sets so delegated to signers.

      We create a hash of the key signing key of the child domain and this creates the trust from parent to child.

      And we follow that hierarchical trust all the way down to the actual record sets.

      And this is how we can trust the contents of all the TLVs.

      This is how the TLVs trust any domain zones.

      And this is how we can trust the contents of any record sets within those domain zones.

      And all of this can be cryptographically validated at every step.

      And this is one of the major benefits that DNSSEC provides.

      Now this point has everything I wanted to talk about in this video.

      In the next video I'm going to be detailing exactly how the key signing ceremony works and why this is such a critical event for the security of DNS using DNSSEC.

      But at this point you can go ahead and complete this video.

      And when you're ready, I look forward to you joining me in the next. in the next.

    1. Welcome back.

      In this video, I want to step through how DNSSEC works inside his own, specifically how it allows a DNSSEC resolver or client to validate any resource records within his own.

      This video is focusing on the data integrity part of DNSSEC.

      And coming up after this is another video where I'll cover the chain of trust and origin authentication benefits which DNSSEC provides in a lot more detail.

      Now, because this video covers digital signing within DNSSEC, it's important that you've watched my previous videos on DNS, on hashing, and on digital signing.

      If you haven't, all of those videos will be linked in the description.

      If you have, then let's jump in and get started.

      Now, to understand DNSSEC, I first need to introduce a term, and that's a resource record set or RR set.

      Let's look visually at what this is.

      We'll start with the DNS zone of ican.org, and I'm using this as an example in this lesson as it's one which I know to be DNSSEC enabled.

      Now, inside his zone, we have a number of resource records.

      First, www.ican.org, which is a CNAME record.

      And remember, CNAMES point at other records.

      And in this case, it points to two other records.

      One of them is an A record, so IP version 4, and the other is an AAA record, which is IP version 6.

      Now, finally, we have four MX records for the domain pointing at four different mail exchange servers.

      Now, each of these are resource records.

      I'm showing a total of seven.

      So what's a resource record set?

      Well, a resource record set or RR set is any records of the same name and the same type.

      So in the case of the left three, this means each of them is their own RR set.

      But in the case of the MX records, they all have the same name, so ican.org, and they're all MX records.

      And this means that all four of these are inside one RR set.

      RR sets are just sets of resource records.

      They make it easier to deal with records in groups versus single records.

      And in the case of DNS set, it keeps things manageable in other ways, but more on that in a second.

      Now, this is what an RR set might look like if you actually interact with DNS.

      Notice how all the names are the same and all the types are the same.

      So why do you need to know this?

      Well, because RR sets are used within DNS set.

      Right now, there's no way to tell if any of these resource records are valid.

      DNS set provides this functionality, but it's not individual resource records which are validated by DNS set.

      It's resource record sets or RR sets.

      Now, let's take a look at how this works.

      So DNS allows us to validate data integrity of record sets within DNS.

      It doesn't work on individual records, rather it works on sets of records, RR sets.

      Let's take a look at how.

      So we start with the same ican.org zone, and inside here I'm going to step through one RR set example, the set of four resource records which make up the MX RR set.

      Right now, without DNS set, if a bad actor found a way to change or make you think that these records have changed, then email delivery could, in theory, be redirected.

      DNS set helps prevent that using two features.

      First, we have RR sync, which stores a digital signature of an RR set using public and private pairs of keys.

      This one key pair is known as the zone signing key or ZSK.

      The private part of this key pair is sensitive and it's not actually stored within the zone.

      It's kept offline, it's kept separated, but you need to know that it exists.

      Like any private key, you need to keep this key safe and not accessible from the public domain.

      So once again, an RR sync contains a digital signature of an RR set.

      So we take the RR set which is plain text, we run it through a signing process, let's call this the digital signature atron 9000.

      In reality, it's just a standard cryptographic process.

      This process uses the private part of the ZSK to create a signature, and this is why it's important to keep the private part of this key safe.

      This output, the signature can be stored alongside the plain text RR set in the zone using the same name, but the record type is RR sync.

      Any normal DNS clients will only see the RR set, any DNS set clients will see the RR set and the corresponding RR sync.

      Now this uses digital signing and hashing.

      If the RR set changes, the RR sync has to be regenerated in order to be valid.

      If the RR set changes without a corresponding change to the RR sync, the result is an invalid signature.

      And so you can tell if anything has changed without the approval of the person controlling the private zone signing key.

      And this is because only the private part of the zone signing key can be used to sign RR sets creating an RR sync.

      Assuming that you trust that the private zone signing key is in safe hands, then you know that if there's a valid RR sync for a corresponding RR set, that RR set is in a valid state created by the zone admin, and if it changes, you can tell.

      Now the important question is, how can a DNS client or resolver verify the RR sync?

      For that, there's another piece to the DNS set puzzle.

      We need the public part of the zone signing key to be able to verify signatures or RRs sync created using the private part.

      Lucky for us, public parts of the key pairs aren't sensitive and don't need to be guarded.

      We just need a way to make them available.

      So consider this scenario, we have the same ICANN.org domain.

      We also have a DNS set resolver here at the bottom.

      How do we know it's a DNS set resolver?

      You'll just have to trust me, but it is a super smart resolver.

      Now inside the zone, we have the MX RR set for the ICANN.org zone.

      We also have the MX RR sync, which remember is a signature for the RR set created using the private part of the zone signing key.

      Inside a DNS set enabled zone, you'll find another record, the DNS key record.

      The DNS key record stores public keys.

      These public keys can be used to verify any RRs in the zone.

      DNS key records can store the public key for the zone signing key, so the ZSK, and also a different type of key, the key signing key or KSK.

      But more on this in a second.

      We're going to do this step by step.

      This is what a DNS key record might look like, and because it can store different public keys, there's a flag value.

      A value of 256 means that it's a zone signing key, and a value of 257 means it's a key signing key.

      So the top one here, this is the zone signing key, and it's this value, which is the last piece of the puzzle.

      It means the DNS resolver can take the RR set, which remember is the plaintext part, and using this together with the matching RRsig and the DNS key record, it can verify that both the RRsig matches the RR set, and the signature was generated using the private part of the zone signing key.

      And the result of this is that our DNS-seq-capable resolver at the bottom can verify the RR set is valid and hasn't been compromised.

      Now this is all assuming, and this is a big assumption, that we trust the DNS key, specifically the zone signing key.

      You have to trust that only the zone admin has the private part of the key, and you also have to trust that it's the correct zone signing key.

      If you do, you can trust the RRsig and matching RR set are valid.

      Now a real-world comparison of this would be to imagine if somebody shows you an ID card which has their photo on it.

      The photo ID only proves their identity if you trust the photo ID is real and it was created by a genuine authority entity.

      In humans, this trust is a bit fake, that's why fake IDs are such a problem.

      This isn't a problem with DNS-seq, because as you'll see, we have a chain of verifiable trust all the way to the DNS route.

      The DNS key record also requires a signature, and this means a matching RRsig record to validate that it hasn't been changed.

      The DNS key record, though, is assigned with a different key, the key signing key, or KSK, so as the name suggests, this key isn't used for signing anything in the zone, instead it's used for signing keys.

      So the zone signing key is used for signing everything in a zone, so to create most RRsig records, except the DNS key records, these are signed by the key signing key creating the DNS key RRsig record.

      Now I get it, I've just introduced another type of key, so let's look at how this all fits together within a DNS zone.

      At this point, I want to cover two really important points about DNS-seq.

      First, how does a key signing key fit into all this and why do we have them?

      And second, what mechanism allows us to trust a zone?

      We know that RRsig records let DNS-seq resolvers verify record sets, but how do we know the keys used within a zone themselves can be trusted?

      To illustrate both of these, let's start with the same ICANN.org zone, and then off to the right side, a related area, but containing more sensitive things, this might be a physical key store like a FileSafe or a HSM device.

      The container here to start with is the private zone signing key, and this is used together with an RRset record to create a corresponding RRsig record.

      Then, the public part of this zone signing key is stored in the DNS key record.

      The flag of 256 tells us that it's a zone signing key.

      At this point, I want to pause and take a quick detour.

      If this was all that we had, so the DNS key, we couldn't trust it.

      Somebody could swap out the DNS key record, put in a new public key in there, use the private part of that fake key to regenerate a fake RRsig, adjust the RRset and take over email for this domain.

      We need a way of ensuring the zone signing key is trusted.

      If we didn't have some kind of trust chain, we would need to manually trust every zone within DNS.

      That would defeat the purpose of having a globally distributed system.

      So, the way that this works is that this zone, so ICANN.org, is linked cryptographically to the parent zone, which is .org.

      So, just like with normal DNS where name server records are used in the org zone to delegate to domains such as ICANN.org, the org parent zone also has a way to explicitly state that we can trust ICANN.org as a zone.

      And I'll talk about exactly how this works in the next video.

      For now, I want to focus on this zone.

      Now, if we use a single key, so just the zone signing key, that would work.

      But this would mean if we ever wanted to change the zone signing key, then we would have to involve the .org parent zone.

      Best practice is that we want to be cycling keys fairly often, and doing that where it also requires updates up the chain would become inefficient.

      And so we have this new key pair, the key signing key, or KSK.

      Now, there's a private part in a public heart.

      The private part is used to sign DNS key record sets, which creates an RRSIG of the DNS key.

      And this makes it easy to change the zone signing key used for a zone.

      We just have to regenerate all of the RRSIG records, update the DNS key record set, and then regenerate the RRSIG of the DNS key record set using the private key signing key.

      All of this is inside our zone only.

      It doesn't involve the parent zone in any way.

      We store the public part of the key signing key in the DNS key record set.

      But now we have a new problem.

      How can we trust the key signing key?

      Well, spoiler, it's referenced from the parent zone, in this case .org.

      So remember how I said that the DNS key record set stored both zone signing and key signing public keys?

      Well, this is the main point of trust for the zone.

      This is how trust is conveyed into the zone.

      Because the parent zone links to the public key signing key of our zone, assuming we can trust the .org parent zone, because it references our zone's key signing key, we can trust our zone's key signing key.

      This key signing key signs our zone signing key, and the zone signing key signs all of the RR sets to create RR6.

      We have a chain of trust created between two different layers of DNS, specifically DNSSEC.

      Now, we're at the point now where you should understand how DNSSEC validates things within a single zone.

      How it uses RR6 to validate RR sets, how it uses the DNS key records to get the public keys to deal with that validation using the zone signing key.

      How a key signing key is used to create an RRSIG at the DNS key, which allows the validation of that DNS key record.

      And how the parent domain or parent zone trusts the public key signing key at the child domain or zone.

      Now, you don't know how this trust occurs yet.

      That's something I'm going to be talking about in the next video.

      I've also stepped through why two different keys are needed.

      Using a zone signing key for signing within a zone, and a key signing key for signing that key, that allows an admin split.

      The key signing key can be referenced from the parent zone, while the zone signing key is used exclusively within the zone.

      And this means that a zone signing key can be changed without requiring any changes to the parent zone.

      So what position does that put us in?

      With this functionality using DNSSEC, what have we gained in the way of additional security?

      Well, we can now verify the integrity of data within this specific DNS zone.

      So, we've eliminated the DNS cache poisoning example.

      Assuming we trust the key signing key, and also assuming the key signing key hasn't been changed as part of an exploit, then we can trust all of the information contained within a zone.

      Now, in the next video, I'm going to step you through exactly how DNSSEC creates this chain of trust.

      And this allows a parent zone to indicate that it trusts the key signing key used within a child zone.

      And the same architecture at every level of DNSSEC means that we can create this entire end-to-end chain of trust, which can be verified cryptographically.

      Now, at this point, that's all I wanted to cover in this video.

      In the next video, I'm going to step through how this trust from a parent zone to a child zone is handled within the DNSSEC hierarchy.

      And we'll go through how the query flow works step by step.

      For now, though, go ahead and complete this video, and when you're ready, I look forward to you joining me in the next.

    1. Welcome back and in this video of my DNS series I want to talk about DNSsec.

      Now this is the first video of a set which will cover DNSsec which provides us with a lot of additional functionality and so requires a few dedicated videos.

      Now to get the most from this video it's important that you've watched my previous videos on DNS as well as on hashing and digital signing.

      If you haven't all the videos that you need I'll link in the description and you should watch those first or have equivalent knowledge.

      DNSsec is a secure add-on for DNS, it provides additional functionality.

      In this video I want to set the scene by talking about why we need DNSsec so let's jump in and get started.

      Now I promise this bit will be super quick but I do need to use some bullet points.

      I hate bullet points as much as anyone but sometimes they're just the quickest way to present information so let's go and please stick with me through this first set of important points.

      Now DNSsec provides two main improvements over DNS.

      First data origin authentication and this allows you to verify that the data that you receive is from the zone that you think it is.

      Are the records returned to you from Netflix.com from the real Netflix.com.

      If you're looking at cash results are they really from that original zone?

      Second DNSsec provides data integrity protection so has the data that you receive been modified in any way since it was created by the administrator of the zone.

      So if you have Netflix.com data is that the same un-changed Netflix.com data which the administrator of the Netflix.com zone created.

      Now it does both of these things by establishing a chain of trust between the DNS route and DNS records but it does this in a cryptographically verifiable way where DNS has some major security holes DNSsec uses public key cryptography to secure itself in a similar way to how HTTPS and certificates secure the HTTP protocol.

      It means that at each stage you can trust if a child's zone has the trust of a parent's zone and you can verify that the data contained within that zone hasn't been compromised.

      Now another really critical part of DNSsec to understand before we touch upon why it's needed is the fact that it's additive.

      It adds to DNS it doesn't replace it.

      It's more accurate to think of any queries that you perform as either using DNS on its own or DNS plus DNSsec.

      Conceptually imagine DNS at the bottom with DNSsec led on top.

      Now in a situation where no DNS exploits have taken place it means that the results will be largely the same between DNS and DNSsec.

      Let's say that we have two devices the one on the left is DNS only and the one on the right is DNSsec capable.

      When querying the same DNS name servers the DNS only device will only receive DNS results.

      It won't be exposed to DNSsec functionality and this is critical to understand because this is how DNSsec achieves backward compatibility.

      A DNSsec capable device though this can do more than just normal DNS so it still makes queries and gets back DNS results but it also gets back DNSsec results and it can use these DNSsec results to validate the DNS results.

      Assuming no bad actors have corrupted DNS in any way then this will go unnoticed since the results are the same but consider a scenario where we have changed DNS data in some way.

      So again we have two devices the device on the left is DNSsec capable and the one on the right is standard DNS.

      So the DNS only device performs a query and it thinks it's getting back the genuine website result only it isn't.

      In an exploited environment it will be unaware that the result it gets the queries are bad and so the website it browses through might not actually be the one that it expects.

      With DNSsec the initial query will occur in the same way and even though it's corrupt and will look valid what follows is that DNSsec will verify the result and because public private key cryptography is used together with a chain of trust architecture DNSsec will be able to identify that records have changed or they come from an origin which isn't the one that we're querying.

      Now it's important to understand that DNSsec doesn't correct anything it only allows you to validate if something is genuinely from a certain source or not and if it's been altered or not it doesn't show you what the result should be but in most cases it's enough to know the integrity of something is valid or in doubt.

      Now it might help you to understand one of the common risks posed by normal DNS if we step through it visually so let's do that.

      Consider this architecture we have Bob on the left who's about to perform a query for the IP address of Categorum.io using this resolver server but we also have a bad actor Evil Bob at the bottom and in advance to perform this exploit he performs a query for Categorum.io this begins the process of walking the tree but while that's happening during this process where the resolver is walking the tree to get the true result Evil Bob responds with a fake response it pretends to be the real server so even while the real process is continuing Evil Bob enters false information into the resolver server in the middle and this result is now cash and the cash has been poisoned with bad data this means that when Bob queries for some Categorum.io records he's going to get the poisoned result this result is going to be returned to Bob and the effect of this is that Bob is going to be directed at something that he thinks is Categorum.io but isn't now this is just one way that DNS can be disrupted it's over simplified and there are some protections against it which illustrates how DNS isn't secure it was built during a time period where the internet was viewed as largely friendly rather than the adversarial zone which he now is from a security perspective now this point I just want to switch across to my command prompt and show you how a normal DNS query differs from a DNS sec query so let's go ahead and do that okay so we've moved across to my terminal and I'm just going to go ahead and use the dig utility which is a DNS utility and I'm going to perform a normal DNS query so this is the command dig its face www.icam.org and if I run at this query this is the result that we receive it's this answer section which I want to focus on now just to reiterate this is the query that I performed for this DNS name so in the answer section we have a result for www.icam.org and the result is that it's a C name and the C name points at another DNS record in this particular case www.vip.icam.org so directly below we can see this DNS record so www.vip.icam.org this time it's an A record an A records point at IP version 4 addresses and this is the IP version 4 address which corresponds to this DNS name and this DNS name maps back to our original query now because this is normal DNS we have no method of validating the integrity of this data you can see here that I'm querying this DNS server so 8.8.8.8 and this is not a DNS server that's affiliated with the ICANN organization so this result is not authoritative it's possible that this data is not valid either by accident or because it's been deliberately manipulated now DNSSEC helps us to fix this risk and let me show you how I'm going to start by clearing the screen and then I'm going to run the same query but using DNSSEC and I can do that using this command adding this additional option on the end of plus DNSSEC when I run this I receive both DNS and DNSSEC results so www.icam.org is a C name and it points at this DNS record slightly below it we can see the record that it points at this is an A record and just as with the previous query results it points at this IP version 4 address now what you'll notice is for each of these normal DNS results we also have this RRSEC and this is a DNSSEC resource type this is basically a digital signature of the record that it corresponds to and I'll show you in the next video how this digital signature can be used to validate the normal DNS data that's stored within this zone so we can query for normal DNS results and then validate the integrity of those results using DNSSEC now in this part of this lesson I just want to demonstrate exactly how a DNSSEC query result differs from a normal DNS result in the next video I'm going to expand on this and set you through exactly how these signatures work within a DNS zone at this point let's move back to the visual okay so what I just demonstrated is a way to avoid this kind of attack because even if a cache was poisoned a DNSSEC capable resolver would be able to identify the poison data and that alone is a huge improvement over standard DNS so at this point I hope you have a good idea of some of the ways which DNSSEC improves normal DNS that's it Identify the poisoned data and that alone is a huge improvement over-sanded at DNS.

      So at this point I hope you have a good idea of some of the ways which DNSSEC improves normal DNS.

      That's it for this video and the next one we're going to explore exactly how DNSSEC works in detail.

      Now it's a lot to get through so I wanted to make sure that each different area of DNSSEC functionality has its own dedicated video.

      At this point though, thanks for watching, go ahead and complete this video and when you're ready I look forward to you joining me in the next.

    1. Welcome back and in this fourth part of this video series I want to cover how a domain is registered.

      So what entities are involved and exactly how this new domain is integrated with DNS.

      So let's jump in and get started.

      The process of registering a domain includes a few key entities.

      First, the person registering the domain.

      Secondly, the domain registrar.

      Examples of this might include Route 53 or Hover.

      Next is the DNS hosting provider.

      Examples again include Route 53 or Hover.

      We have the TLD registry.

      In the case of the .com/tld this is Verisign and then finally the .com/tld zone managed by the same entity.

      Now one really important and often confusing element to this is the difference between the registrar and the DNS hosting provider.

      These are different things, different functions.

      The registrar has one function to let you purchase domains.

      And to allow this they have a relationship with the TLD registry for many top level domains.

      You might use Hover for example to purchase .com domains, .io domains, .org domains and so on.

      And for each of these they will communicate with a different TLD registry.

      So the registrar lets you register i.e. purchase domains.

      A DNS hosting provider they operate DNS name servers which can host DNS zones and they allow you to manage the content of those zones.

      Now why this is confusing is that some companies are only registrars, some companies are only DNS hosting providers but some like Hover and Route 53 can do both.

      If you know AWS if you work with Route 53 registered domains area of the console then this is the domain registrar function.

      If you work in the hosted zones area this is the DNS hosting provider area.

      Try and think of them as two separate things because it makes the explanation of domain registration much more logical.

      Step number one in the domain registration process assuming the domain is available is that we pay for the domain via the domain registrar.

      Examples include GoDaddy, Route 53, Hover and many more.

      It's at this point that we're going to need a DNS zone for the domain being registered and this zone needs to be hosted on some DNS name servers.

      So if the DNS hosting provider is the same company as the registrar the zone is created and hosted automatically.

      If it's a different company you'll be asked for the name server information where the zone is hosted already and this has to be configured separately.

      So at this point we have a domain being registered.

      We have a DNS zone ready to go hosted on some name servers and we have all the networking information for those name servers.

      So next the registrar communicates this to the registry for the TLD.

      In the case of the .com TLD this is Verisign.

      Now next Verisign assuming everything is good they add all of those details to the .com TLD zone and at this point the domain is live.

      For a domain to be live the name servers which host the zone need to be pointed at from the relevant TLD zone.

      If this zone ever changes for example if it's moved to different name servers the entries in the TLDs or the NS records pointing at the name servers for this domain they need to be changed and this is how a domain is registered.

      The key point really is to understand the two different roles the domain registrar who register the domain with the registry and the hosting provider who host the zone for the domain on name servers.

      Many companies do both but they are conceptually different.

      At this point thanks for watching that's everything I wanted to cover about registering a domain so go ahead and complete this video and when you're ready I look forward to you joining me in the next.

    1. Welcome back and in this video, the third in the series, I want to talk about how DNS works and going to cover the structure and the flow of making a DNS query.

      So let's get started.

      Now I want to quickly touch upon the core functionality that DNS brings.

      If we abstract away from all the technical details for a second, we have a person, device or service, it has a DNS name and it needs an IP address which provides services for that name.

      So in this example we have www.netflix.com and we need the IP address or addresses which we can connect to in order to access Netflix.

      So we're in the world, there is a DNS zone for Netflix.com which has the answer that you need.

      It contains a record which links www.netflix.com to one or more IP addresses.

      The issue is how do we find this zone?

      Well that's what DNS does.

      It's the job of DNS to allow you to locate the specific DNS zone which can provide you with an authoritative answer.

      So you can query it and be provided with a response.

      The IP address or addresses in this example which provide the services www.netflix.com.

      Everything we talk about next, it's all part of the process to allow you to find the correct zone.

      So DNS is a huge global distributed database containing lots of different DNS records and the function of DNS is to allow you to locate the specific zone which can give you an authoritative answer.

      So let's step through how a query works within DNS.

      In this example imagine that we're sitting at a computer and we're querying the www.netflix.com.

      The first thing to be checked will be the local DNS cache and host file on the local machine.

      The host file is a static mapping of names to IPs and overrides DNS and so it's checked first.

      Then the machine and potentially the application being used might have some local DNS caching and so that's checked before we proceed with the query.

      Assuming that the local client isn't aware of the DNS name that we're querying this is where we move on to the next step which is where we use a DNS resolver.

      A resolver is a type of DNS server often running on a home router or within an internet provider and it will do the query on our behalf.

      So we've sent the query to the resolver and now it takes over.

      Now from the resolver's point of view first it also has a local cache which is used to speed up DNS queries and so if anyone else has queried www.netflix.com before it might be able to return a non-authoritative answer and remember it's non-authoritative because the only server which can give an authoritative answer is the one which is pointed at four given domain from the TLD zone of that domain.

      Now since the resolver isn't that while it can cache results it will always return them as non-authoritative.

      In most cases nobody will care but it's important to understand the distinction.

      Now let's assume that there's no cache entry for www.netflix.com.

      Well the next step is that the resolver queries the root zone via one of the root servers.

      Every DNS server will have these ID addresses hard coded and this list is maintained by the operating system vendor.

      The DNS root won't be able to answer us because it isn't aware of www.netflix.com but it can help us get one step closer.

      The root zone contains records for dot com specifically name server records which point at the name servers for the dot com TLD.

      This is how trust is created and how the root zone delegates control at the dot com TLD to verisign.

      So the root servers will return the details at the dot com name servers.

      Now this isn't exactly what we're looking for but it does bring us one step closer.

      So the resolver can now query one at the dot com name servers for www.netflix.com.

      Assuming that the netflix.com domain has been registered the dot com zone will contain entries for netflix.com.

      This is how verisign delegates control of the netflix.com domain when the domain is registered and so the dot com name servers while they can't get the resolver the answer that it needs they can help it to move one step closer and so the details of the netflix.com name servers are returned to the resolver.

      Well now the resolver can move on and so it queries the netflix.com name servers for www.netflix.com.

      Because these name servers are authoritative for this domain because they host the zone and zone file for this domain and they're pointed at by the dot com TLD zone they can return an authoritative result to the query back to the resolver.

      Now the resolver caches the result in order to improve performance for any of the same queries in future and it returns this result through to the client which is our machine.

      This is how every DNS query works so maybe quicker if they're cached or longer if the full walking the tree process needs to occur.

      The key facts to keep in mind are that firstly no one single name server has all the information not even the root name servers but and this is key to how DNS operates every query to every name server moves you one step closer to the answer.

      The root gives you the dot com name servers the dot com name servers gives you a netflix.com name servers and the netflix.com name servers will be able to give you an authoritative result and this process end to end is called walking the tree.

      Now this is the process from a high level but technically how does it look well let's check that out.

      We start with the root zone and I've skipped a few steps to step four when the resolver is querying the root zone.

      The root zone doesn't have the information needed but it does know which nameservers handle dot com and so it can provide this information but these are a subset of the name servers run by Verisign which manage the dot com TLD.

      So these are the servers which host the dot com zone file so we can now query the dot com zone.

      We can't get the answer directly from here but it does know which name servers are authoritative to netflix.com so these are the network addresses of the servers which host the netflix.com zone and this is authoritative so this will give us the answer that we need but in this case it's not an IP it's another DNS name this is a CNAME record and I'll talk about these in another video but what this means is that to get the IP address for this we have to follow the same process through again that's right many queries like this end with another DNS name which requires another query.

      One of the many reasons an application can perform badly is if many DNS calls are used within the application and network performance impacts these in a negative way so this is how the walking the tree process works end to end this is how every DNS query works when you're trying to look up the IP address for a given DNS name this architecture and this flow is at the core of how most internet-based applications work from a DNS perspective and it's a really critical set of knowledge to understand fully but at this point that is everything I wanted to talk about in this video thanks for watching go ahead and complete the video and I look forward to you joining me in the next.

    1. Welcome back, and in this video, which is part two of my DNS mini-series, I want to cover some of the reasons why DNS is structured in the way that it is.

      Why do we need lots of DNS servers?

      Why isn't one enough?

      Additionally, at the end, I'm going to introduce some key DNS terms, and then introduce the hierarchical structure of DNS.

      So let's jump in and get started.

      Now, there are a few main problems with just having one DNS server, or even a small number of servers.

      It's important that you understand these reasons, because it will help you understand why DNS is architected in the way that it is.

      First, there's the obvious risk problem.

      A small group of bad actors could attack the DNS infrastructure, and without much effort, prevented servicing genuine requests.

      And that's something that we have to avoid for critical systems on the internet.

      Also, we have a scaling problem.

      Almost everyone who uses the internet globally uses DNS.

      This represents a massive and growing load on the system.

      A single server or a small group of servers can only get so big.

      If every access is being made against one or a small group of servers, no matter what information is being requested, the system cannot scale.

      Now, additionally, DNS is a huge database.

      Current recent estimates predict that there are around 341 million domains, such as Netflix.com, Apple.com, and Twitter.com.

      And each of those domains might have many records, tens, hundreds, thousands, or even more records in each of those domains.

      And so this represents a huge data volume problem.

      We can start with the amount of data, but then also need to take into consideration updates to that data, as well as consistency issues.

      And all of that data is accessed by anyone using the internet on a constant basis.

      Now, we can address the risk problem by creating more servers, each of them storing an exact copy of the same DNS data.

      The more servers we have, the more load can be tolerated against those servers, and the less risk of attackers managing to take down the entire platform.

      But this method doesn't really do anything about the scaling problem.

      If every user of DNS communicates with any of the servers at random, it means that every server needs to hold the complete DNS dataset.

      And this is a huge amount of data.

      It's a huge global scale monolith, and this is something that we need to avoid.

      Ideally, we also need to have the ability to delegate control over certain parts of the DNS dataset to other organizations so that they can manage it.

      So UK domains, for example, should be managed by a UK entity.

      US domains should be managed by somebody in the United States.gov, by the US government.au, by an organization in Australia, and so on.

      And for this, we need a hierarchical structure.

      I'm going to be talking about this fairly soon, but for now, I need to introduce some DNS terms.

      Now, I need you not to switch off at this point in the video.

      I'm going to be using some bullet points.

      I hate bullet points, but these are worth it, so please stay with me.

      The first term that I want to introduce is a DNS zone.

      Think of this like a database.

      So we have Netflix.com, and that's the zone.

      Inside that zone are DNS records, for example, www.netflix.com, as well as many others.

      Now, that zone is stored on a disk somewhere, and that's called a zone file.

      Conceptually, this is a file containing a zone.

      So there's a Netflix.com zone file somewhere on the internet containing the Netflix.com zone and all of its records.

      We also have DNS nameservers, known as NS for short, and these are DNS servers which host one or more zones, and it does so by storing one or more zone files.

      It's the nameserver or nameservers of the Netflix.com zone, which can answer queries that you have about the IP address of www.netflix.com.

      Next, we have the term authoritative, and this just means that for a given domain, this is the real or genuine, or to put another way, the boss for this particular domain.

      So there are one or more nameservers which can give authoritative answers for www.netflix.com.

      These can be trusted.

      They're the single source of truth for this particular zone.

      And we also have the opposite of this, which is non-authoritative or cached, and this is where a DNS server might cache a zone or records to speed things up.

      Your local router or internet provider might, for instance, be able to provide a non-authoritative answer for learn.control.io or youtube.com, because you've visited those sites before.

      But only my nameservers can give an authoritative answer for learn.control.io.

      Now that you know those terms, I want to introduce the architecture of DNS, so it's hierarchical structure.

      And don't worry, this is high level only.

      I'll be covering the detail of how DNS works and how it's used in a follow-up video.

      So at this point, you know why a single DNS server is bad.

      You also know why having many DNS servers is bad if they just saw the same monolithic set of data, and you understand a few important DNS terms.

      What I'm going to step through now is a hierarchical design, which is the way that DNS works.

      Using this architecture, you can split up the data which DNS stores and delegate the management of certain pieces to certain organizations.

      Splitting the data makes it easier to manage and also splits the load.

      If you're doing a query on netflix.com, you generally won't have to touch the infrastructure at the twitter.com.

      Now DNS starts with the DNS root, and this is a zone like any other part of DNS, and this zone is hosted on DNS name servers also just like any other part of DNS.

      So the DNS root zone runs on the DNS root servers.

      The only special element of the root zone is that it's the point that every DNS client knows about and trusts.

      It's where queries start at the root of DNS.

      Now there are 13 root server IP addresses which host the root zone.

      These IP addresses are distributed geographically, and the hardware is managed by independent organizations.

      The internet corporation for assigned names and numbers, or ICANN, operates one of the 13 IP addresses which host the root zone.

      And others include NASA, the University of Maryland, and Verisign.

      So to be clear, these organizations manage the hardware for the 13 DNS root server IP addresses.

      In reality, each of these 13 IP addresses represents many different servers using anycast IP addresses.

      But from DNS' perspective, there are 13 root server IP addresses.

      Now the root zone, remember, this is just a database.

      This is managed by the internet assigned numbers authority known as IANA.

      So they're responsible for the contents of the root zone.

      So management of the root zone and management of the root servers which host the root zone is different.

      Now the root zone doesn't store that much data.

      What it does store is critical to how DNS functions, but there isn't that much data.

      The root zone contains high level information on the top level domains or TLDs of DNS.

      Now there are two types of TLD, generic TLDs such as .com and country codes specific ones such as .uk and .au.

      IANA delegates the management of these TLDs to other organizations known as registries.

      Now the job of the root zone really is just a point at these TLD registries.

      So IANA delegates management at the .com TLD to Verisign, meaning Verisign is the .com registry.

      And so in the root zone, there's an entry for .com pointing at the name servers which belong to Verisign.

      Now .com is just one TLD.

      There are other entries in the root zone for other TLDs and other TLDs could include .io, .uk and .au and many more.

      Because the root zone points at these TLD zones, they're known as authoritative the source of truth for those TLDs.

      This process where the root zone points at the name servers hosting the TLD zones, it establishes a chain of trust within DNS.

      So to summarize, the root zone is pointing at the name servers hosting the TLD zones run by the registries which are the organizations who manage these TLDs.

      So Verisign will operate some name servers hosting the .com TLD zone and the root zone will have records for the .com TLD which point at these .com name servers.

      The .com zone which is just another DNS zone also contains some data, specifically high level data about domains which are within that .com TLD.

      For example, the .com TLD zone contains some records with Twitter.com and Netflix.com, so records for domains which exist inside the .com zone.

      The TLD only contains this high level information on domains within it, for example, Netflix.com.

      It doesn't contain detailed records within these domains, for example, www.netflix.com, all the TLD contains is information on the domain itself.

      Specifically, with this example, a set of records for Netflix.com which point at the name servers which host the Netflix.com zone.

      Now it will also contain records for Twitter.com which point at the name servers which host the zone for Twitter.com as well as records for every other domain within the .com TLD.

      Now these name servers, because they're pointed at from the layer above, they're authoritative for the domains, the zones that they host.

      So the name servers for Netflix.com are authoritative for Netflix.com because the Netflix.com entry in the .com TLD points at these name servers.

      Now these name servers host the zone for a given domain, for example, Netflix.com.

      This means the servers host the zone file which stores the data for that zone.

      At this level, the zone contains records within Netflix.com, so www.netflix.com which points at a set of IP addresses.

      And because the zone and zone files are on these name servers and because these name servers are authoritative for the domain, these zones and zone files are also authoritative.

      Now don't worry about understanding this in detail.

      In the next video, I'm going to be walking through how this works in practice.

      For now, all I need you to understand is that each layer of DNS from the root, the TLDs and the domain name servers they store.

      [POP] Or only a small part of the DNS database.

      The root zone knows which name servers the .com zone is on, the .com zone knows which name servers Netflix.com zone is on, and the Netflix.com zone contains records for the Netflix.com domain and can answer queries.

      So this is the hierarchical architecture of DNS.

      And in the next video, in this video series, I'm going to be stepping you through the flow of how DNS works and discussing the architecture at a more technical level.

      But at this point, that's everything I'll be covering in this video.

      So go ahead and complete the video.

      And when you're ready, you can join me in the next video of this series.

    1. Well, welcome to the first video in this series where I want to help you understand DNS.

      DNS is one of the core services on the internet, and it doesn't work, applications and other services will fail.

      Now in this video series I'll be covering what DNS does, why it's structured the way that it is, how DNS works to get us answers to queries, and I'll finish up by covering some of its key limitations.

      Now with that being said, let's jump in and get started.

      Now before I cover how DNS works and why it works in the specific way that it does, I want you to be 100% sure of what functionality DNS provides.

      Now when you access any website, you type the name into your browser, for example www.netflix.com.

      Now you might imagine that the name is used to connect to the Netflix.com servers and stream your movie or TV show, but that's not actually how it or any internet app generally works.

      Simply put, humans like names because they're easy to remember, but networks or servers not so much.

      To communicate with Netflix, your computer and any networking in between needs the IP addresses of the Netflix servers.

      DNS actually does many different things, but at its core it's like a big contact database.

      In this context, it links names to IP addresses, so using DNS when accessing Netflix, we would ask DNS for the IP address of Netflix.

      It would return the answer and then our device would use that IP address to connect over the internet to the Netflix servers.

      So conceptually, the main piece of functionality which DNS provides is that it's a huge database which converts DNS names from Netflix.com into IP addresses.

      Now so far, I hope that this makes sense.

      At this point, it sounds just like a database and nothing complex, and you might be asking yourself why not just have one DNS server globally or a small collection of servers?

      Now we're going to review that in the net.

      Next video.

      For this video, I just wanted to set the scene and make sure you understand exactly what functionality DNS provides.

      DNS is critical.

      Many large-scale failures on the internet are caused by either failed DNS or badly implemented DNS infrastructure.

      If you want to be effective at designing or implementing cloud solutions or network-based applications, you have to understand DNS.

      So if you are interested in knowing more, then go ahead and move to the next video, where I'll cover why DNS is structured in the way that it is.

    1. Welcome back and in this lesson I want to talk about digital signing or digital signatures.

      This is a process which you need to be familiar with to understand many other areas of IT such as DNFSEP or SSL certificates.

      If you haven't watched my video on hashing it's linked in the description and you should pause this video and go and do that now because it's required knowledge for this video.

      At this point though let's jump in and get started straight away.

      Now before I cover how digital signatures work in detail I want to give you a quick refresher on public key cryptography.

      With public key cryptography you generate two keys as part of a pair.

      We have on the left the private key and this part is secret and should only ever be known by the owner.

      Then we have the public key and this is not secret.

      In fact anyone can know it or possess it.

      It should be public and ideally as widely distributed as possible.

      Now these keys are related, they're generated as part of one operation.

      It means that you can take the public key which remember anyone has or anyone can get and use it to encrypt data which can then only be decrypted by the matching private key.

      The data can't even be decrypted using the same public key which encrypted it so it allows you to securely send data to the owner of the private part of the public key.

      What this architecture also allows is that you can take something and sign it using the private key and think of this just like encryption but where anybody who receives the data and has the public key can view the data and verify that the private key part was used to sign it and this provides away the evidence that you are in control of the private key used to sign some data since the private key is secret and since only you should have it.

      This provides a way to establish a form of digital identity or digital validation.

      This signing architecture is important because it forms the foundation of many other IT processes.

      The process of adding digital signatures to data adds two main benefits when used in conjunction with hashing.

      It verifies the integrity of the data that the data you have is what somebody produced and it verifies the authenticity so the data that you have is from a specific person.

      So integrity is what and authenticity is who.

      Together it means that you can download some data from Bob and verify that Bob produced the data and that it hasn't been modified.

      Nobody can falsify data as being from Bob and nobody can alter the data without you being able to tell.

      Now a key part of this process is that it's led on top of normal usage so if somebody doesn't care about integrity or authenticity they can access data as normal without any of these checks and to enable that the first step is to take a hash of the data that you're going to be validating.

      The original data remains unchanged in plain text or whatever its original format is.

      If you don't care about the integrity and authenticity then you can use the application or consume the data without worrying about any of this process.

      Now this means that anybody having both the data and the hash knows that the data is the original data that Bob produced assuming that they trust the hash to be genuine and that's what digital signatures enable.

      Next Bob signs the hash using his private key and this authenticates the hash to Bob.

      Bob's public key can be distributed widely onto many locations that Bob controls and using this public key you can access the signed hash.

      Because of this you know it came from Bob's private key i.e.

      Bob and so the hash is now authenticated as being from Bob.

      Nobody can falsify a fake hash because only one person Bob has Bob's private key.

      So now we know the hash is from Bob we can verify that because we have Bob's public key.

      We know the hash can't be changed because only Bob has Bob's private key and only the private key can sign anything and appear to be from that private key.

      We now have this authenticated hash and we verified the integrity because of the private-public-key relationship.

      We also have the original document and we know that it's authentic because if it was changed then the hash wouldn't match anymore.

      So if we trust Bob's public key then we know that anything being signed by his private key is authentic because we know that Bob and Bob only have this private key then we trust the entity i.e.

      Bob and because Bob can now digitally sign a hash we know that any data that we receive from the Bob is both authentic i.e.

      Bob has authored this data and it's not been changed during transit or download.

      So we have this chain of trust and this chain of trust using public key cryptographic signing and hashing forms a foundation and many things within id which you take for granted.

      Okay so now that you understand the basic building blocks of this process let's step through this visually.

      So step one is Bob and his private key.

      Bob is the only person to have his private key but he's uploaded his public key to his website, his twitter account, various public key retro services and he includes a link to his public key in all of the emails that he sends.

      Now if you look at all of these copies of his public key if all of them match then you can assume that they're valid and from Bob.

      To exploit this you have to take over all of these services at the same time and change all mentions of his public key so the wider the distribution of the public key the easier it is to spot if any of them have been modified.

      So let's say that Bob creates a contract to send it to a business partner and he wants others to be able to verify firstly that he sent it and that he wasn't altered in transit.

      So the next step is that he puts the document through a hash function which results in a hash.

      The hash as you've learned is unique to this document.

      Know all the document can have this hash and any changes to this document also change this hash and you can't derive the document from the hash.

      What we can't prove yet is that Bob created this hash or that the hash hasn't changed as part of faking the document.

      So to fix this Bob uses his private key and he signs the hash and this creates a signature and he bundles this signature with the original document which creates a signed document.

      So this signed document is the original document so the data plus a copy of the signed hash of that document and that data is going to be hosted somewhere so let's say that Bob uploads it to the internet or he emails the document to somebody or stores it from some cloud storage.

      Now next one of Bob's business partners downloads the contracts of the signed data and the first thing is to get Bob's original hash and so to do that we take the signature and Bob's public key and that gives us back Bob's hash.

      So now we know that this hash is signed by Bob and we know that Bob created this hash and this means that we know what Bob thought the hash of the document was.

      We know the original state of the document when Bob generated the hash.

      So we take the document and we hash it with the same hash function as Bob used.

      So now we have our hash of the document and we have Bob's original hash verified to be from Bob.

      Now these two hashes match you know that the document that you have is the document that Bob generated.

      It hasn't been altered and you know that it originated from Bob because his hash was signed using his private key to generate the signature which is part of this document which is digitally signed and this is how hashing together with public key cryptography specifically signing can be used to verify authenticity and integrity.

      Now Bob could have taken this a step further and encrypted all of this with a public key at the intended recipient and ensured that all of this process could happen in a fully encrypted way but encryption and signing are two slightly different things which are both enabled by the same public key cryptography architecture.

      Now this point has everything I wanted to do.

      I just wanted to give you a really high-level overview of how digital signatures can be used to verify both the integrity and the authenticity of data.

      Now you're going to be using this knowledge as you learn about all that important IT processes so it's really important that you understand this end-to-end but at this point that is everything I wanted to cover in this video so go ahead and complete the video and when you're ready I look forward to you joining me in the next.

    1. Welcome back and in this lesson I want to talk about hashing, what it is and why we need it.

      Now we do have a lot to cover so let's jump in and get started straight away.

      What's simply hashing is a process where an algorithm is used to turn this or any other piece of data into this, a fixed length representation of that data.

      Hashing is at the core of many critical services that you use today, such as passwords, digital signatures, SSL certificates, data searches, and even some antivirus or anti-malware solutions rely on hashes so they can store definitions of malicious files without having to store every single huge file.

      We even have some forms of digital money such as Bitcoin which use hashing.

      Now to understand why hashing is so important we need to step through how it works, what benefits it provides and some of the terminology used.

      So let's go ahead and do that.

      Hashing starts with a hash function and think of this as a piece of code and algorithm.

      Now there are many different ones which are used or have been used over the years.

      An example include MD5 and SHA-2 256.

      The core principle of hashing is that you take some large variable sized data and you put that data through a hashing function and receive a fixed size hash of that data.

      So whether the data is a text file, an image or a huge database file, the hash you receive will be tiny and fixed length based on the hashing function type.

      Now also critically if you take some other data and again put it through the same hashing function you will get a different hash, a unique hash value.

      Even if the data differs by only one byte, one character or one pixel it will result in a different hash.

      The aim with hash functions is that any change no matter how minor will result in a different hash value.

      Another critical part of hashing is that while getting a hash from some data is trivial what you cannot do is take a hash value and get the original data back.

      There's no way to do this.

      Let's say that you had a hash of an image.

      Well you couldn't take the hash and derive the image.

      You could if you had infinite processing power and infinite time brute force every image in existence to try and link back to the hash.

      But this would require hashing every single image until you found the correct one.

      You should view it as impossible with any modern hashing algorithm to derive the original data from a hash.

      Without some vulnerability in the hashing function or without infinite time and processing power the hashing is one way only.

      Data through to hash.

      Now lastly another fundamental part of hashing is that given the same data you're always going to get the same hash value if you use the same hashing function.

      So for one piece of data you get one hash value, for a different piece of data you get a different hash value and hashing is one way only.

      And you should never get the same hash value for different data.

      There are more on this in a second.

      Let's look at an example of where hashing can be used which should be pretty familiar.

      Imagine you're using an online service and they have a single server.

      And when you create an account on that service you create a username and password and both of those are stored on that server.

      When you log in you send your username and password to the server and an automated process checks the password that you send with the one that's stored and if they match you can log in.

      Now even if you encrypt your password in transit and even if the password is encrypted when stored on the server your password still exists on that server.

      And it means if the server is ever exploited then a bad actor will have access but worst to your password and at best an encrypted version of your password.

      Assuming a full data dump or a long term exploit is pretty trivial to get the plain text version of your password in this way.

      If you use that password on all the services those services are also at risk.

      But what about if we use hashes?

      Well with this architecture instead of sending a password to the server when signing up or signing in we send a hash of the password.

      This means the server instead of having our actual password it only stores the hash of our password.

      All the server needs to do is check that the hash that you send matches the one in its database and it can confirm the password was entered correctly on the local client.

      Because given the same data in this case password and the same hashing algorithm you'll end up with the same hash value.

      So by comparing the hash value it stores to the hash value that you deliver by the hash of your password it can check you're entering the correct password without ever storing a copy of your password.

      Now in this example I've used the MD5 hashing algorithm but as you're going to say in a second this isn't super secure anymore so you'd likely need to use another hashing algorithm.

      I'm going to use MD5 as an example to demonstrate a weakness of some algorithms.

      So to stress you wouldn't generally use MD5 for production password systems you'd use something a lot more secure.

      Now if this server was ever exploited when using password hashes this would be much safer because the hashes are one way you can't derive the password from a hash.

      Nothing stops the attacker though from getting over and over again trying every possible word and phrase combination with the hashing algorithm used until it gets us right.

      And then it has confirmed the password that you used and it can try and exploit all the services which you also make use of.

      And this is why it's really important to use a modern secure hashing algorithm.

      Now there are two things with hashing which are really bad.

      One is if we were ever able to take a hash and derive the original data but as I mentioned earlier that's basically impossible to have a critical vulnerability in how hashing works.

      Another major problem would be a collision.

      An example of a collision is that if we take this image of a plane and if we hash this image and then if we take another image say this image of a shipwreck and we also hash this image we should have two different hash values.

      If we do awesome.

      If not if the hash of A on the left equals the hash of B on the right then bad things happen because we can no longer trust the hash function i.e. the hash algorithm.

      And this is one of the reasons that MD5 has hashing algorithm is less trusted because collisions can happen.

      We can actually show how they can be created, how data can be manipulated to cause collisions.

      Now I've attached some links to this video with the research project showing how we can create those collisions.

      But as a quick example I want to switch to my terminal and demonstrate how this works with these two images.

      So in this folder on my local machine I have two images plane.jpg and ship.jpg and those represent the two images that you've just seen on your screen moments ago.

      I'm going to go ahead and generate a hash value of one of these files and I'm going to use the MD5 hashing algorithm.

      So I'm going to put MD5 space and then plane.jpg.

      Go ahead and focus on the hashing value that's been generated so this hashing value in theory should uniquely represent the image plane.jpg.

      So plane.jpg should always generate this hash value and if I repeat this command I get the same hash value.

      But what should happen is if I generate a hash of another piece of data I should get a different hash value.

      So I'm going to run MD5 again this time on ship.jpg.

      Watch what happens in this case it's the same hash value and this is an example of a collision where two different pieces of data generate the same hash value.

      And this is a weakness of MD5.

      We can create this collision.

      We can adjust the data in order to generate this collision and this is a bad thing.

      This shouldn't happen.

      Now I'm not to follow the same process but using a more secure hashing algorithm.

      So this time I'm going to use the SHA-2 256 algorithm on the same file plane.jpg.

      Now watch what happens now.

      The hash value is longer because this is a different hashing algorithm but we confirm that this hash value is for plane.jpg.

      Now I'm going to run the same hashing algorithm on ship.jpg.

      This time note how it's a different hash value.

      This is a much more secure hashing algorithm.

      SHA-2-256 is much better at protecting against these collisions.

      It's a more modern and more well trusted hashing algorithm.

      Now just like any other form of security such as encryption it's important that you always use the latest and most secure protocols and architectures.

      So if you're using hashing in production you should generally use something like SHA-2-256 because if you want to guarantee that one-to-one link between a piece of data and a hash so that any other piece of data generates a different hash you need to make sure you're using a well respected hashing algorithm such as SHA-2-256.

      Now the likelihood of this happening in normal usage is nearly possible because these two images have actually been artificially adjusted to cause this collision but it does represent theoretical vulnerability in the MD5 hashing algorithm.

      I've included links attached to this video which detail the search project and some examples of how you can implement this as a personal project if you want.

      But at this point I'm going to go ahead and return to the remainder of this video.

      Now just to summarise with hashing you take some data plus a hashing function and you generate a hash value.

      Now you can't get the original data from a hash it is a one-way process and the same data should always generate the same hash.

      Different data should always generate a different hash.

      Now just demonstrated how you can artificially cause a collision using older hashing algorithms but in the real world even older algorithms should generate a different hash for different data and any modern hashing algorithm is protected against even this artificial process.

      Now hashing can be used to verify downloaded data.

      If you're making some data available to download you can have the download in one location and the hash of that download stored on a different system.

      It means that you can download the data you can hash it and generate your own hash value.

      If the hash value is matched then the downloaded data hasn't been altered.

      If they differ then it means that what you have is not the same data as what was made available by the original author.

      And this is a process that's very often used to verify sensitive applications.

      So if you're downloading any form of application which stores sensitive data or operates on sensitive networks then you'll generally find that a hash will be made available by the author of that application and it can be used to verify that that download has not been adjusted.

      Now in these type of security sensitive situations or if you're a security professional you also need to be sure that the hash itself hasn't been altered.

      It's also whether the hash itself was generated by the person who claims to have generated that hash.

      So if I make some software available to you and you download it you need to first check that the download hasn't been altered by hashing it yourself and comparing your hash to the hash that I publish.

      But you also need to be sure that it was me publishing that hash and that the hash that you download hasn't been altered in some way.

      And a way that this can be done is using digital signing or digital signatures and this is something that I cover in another video.

      But at this point that's everything I wanted to cover in this video so go ahead and complete the video and be ready.

      I'll look forward to you joining me in the next.

    1. Now another process which uses asymmetric keys is signing.

      Let's review an example.

      The robot general wants to respond to the battle plan.

      So let's say that the robot general has received the battle plans from the cap ruler, and he wants to confirm A that he's received them, and also that he agrees to them.

      Remember, the battle plans require both sides to operate as one, and so the cap ruler needs to know that the robot general has received the plans, and that he agrees with them.

      The general might want to respond with a simple OK message.

      So we'll send the message saying OK to the cap ruler.

      The issue is that anyone can encrypt messages to another party using asymmetric encryption.

      Your eye could get hold of the cap ruler's public key and encrypt a message saying OK, and send it to the cap ruler.

      And the cap ruler wouldn't necessarily be aware whether that was from the robot general or not.

      Just because the cap ruler gets a message from what appears to be the robot general saying OK, it doesn't mean that it's actually from the robot general.

      It could be from a human pretending to be the robot general.

      Encryption does not prove identity.

      But what we can use is a process called signing.

      With signing, the robot general could write this OK message, and then he could take that message, and using his private key, he can sign that message.

      Then that message can be sent across to the cap ruler.

      And when the cap ruler receives that message, he can use the robot general's public key to prove whether that message was signed using the robot general's private key.

      So this is the inverse.

      On the previous example, I demonstrated how you can use the public key to encrypt data that can only be decrypted with the private key.

      In this example, we can take the robot general's private key and sign a document.

      And then the public key of the robot general can verify that that document was signed using its matching private key.

      At no point is the private key revealed.

      It's just because of this relationship between the public and private key.

      The public key can verify whether its corresponding private key was used to sign something.

      And so signing is generally used to verify identity.

      As long as we can be sure that the public key belongs to the robot general, and generally this is done by the robot general uploading his public key to his Twitter account, or putting it on his cloud storage or his website.

      As long as that verification exists, we can get the public key and verify that a document has indeed been signed by his private key because of that relationship between these two keys.

      So key signing is generally used for ID verification and certain logon systems.

      There's one more thing I want to talk about before I finish up this lesson.

      And that's steganography.

      Sometimes encryption isn't enough.

      The problem with encryption is that if you use it, it's obvious that you've used it.

      If you encrypt a file and deliver it to me, there isn't really any scope for denying that you have encrypted some form of data.

      The government, who control the men and women with guns, can often insist that you decrypt the data.

      And if you don't, well, they have plenty of sticks so they can put you in jail.

      You can refuse to decrypt the data, but the deniability isn't there.

      If you encrypt data, somebody will know that you've encrypted data.

      Now, steganography is a process which addresses this.

      If you've ever used invisible ink, the kind which only shows under a certain type of light or when heated, that's a physical form of steganography.

      It's a method of hiding something in something else.

      With steganography, the cap ruler could generate some ciphertext and hide it in a puppy image.

      The image could be delivered to the robot general who knows to expect the image with some data inside and then extract the data.

      To anyone else, it would just look like a puppy image.

      And everybody knows there's no way that the cap ruler would send the robot channel a puppy image to this plausible deniability.

      The effect of steganography might be a slightly larger file, but it would look almost identical.

      Effective steganography algorithms make it almost impossible to find the hidden data unless you know a certain key, a number, or a pattern.

      Steganography is just another layer of protection.

      The way it works at a foundational level is pretty simple.

      Let's say that we wanted to hide a simple message, just high.

      Well, the decimal values for h and i are 8 and 9.

      So we might take the puppy image and pick two random pixels and change the color by 8 and 9 respectively.

      The steganography algorithm would take the original picture, selects the required number of pixels, adjust those pixels by a certain range of values.

      And what it would generate as an output would be an almost identical puppy image.

      But hidden there would be slight changes.

      If you don't believe me, let's blow this up a little bit because this is an actual simple example of steganography.

      If you look really closely at where those arrows are pointing, the color is slightly different than the background.

      The first pixel has been adjusted by eight values and the second has been adjusted by nine values.

      And so if you knew the location of these two pixels, you could take the second image and extract the text.

      Now, this is a super simple example.

      A real algorithm would be much more complex.

      But this is at base level how the process works.

      It allows you to embed data in another piece of data.

      To be really secure, the cap roller would encrypt some data using the robot general's public key, take that cypher text, use steganography to embed it in an image that wouldn't be tied back to the cap roller, send this image to the robot general, and then the robot general could also use steganography to extract the piece of cypher text and decrypt it using his private key.

      And the same process could be followed in reverse to signal an OK.

      But the robot general, in addition to encrypting that OK, would also sign it so the cap roller would know that it came from the robot general.

      With that being said, go ahead, complete this video.

      When you're ready, you can join me the next.

    1. Welcome to this video where I want to talk at a high level about hardware security modules known as HSMs.

      Now these are a really important type of device to understand both in general and especially if you currently work in or want to work in the security space because so many other things rely on them to function.

      Now let's jump in and step through why we need them, what they are and how they work.

      Now let's start by looking at the world without hardware security modules and we're going to do that with the example of a virtualized environment and so we have a VM host and this could be VMware, Zen or something within a cloud environment.

      It doesn't matter for this example.

      This means that we have some physical CPUs, memory as well as mechanical or solid state storage.

      The reason on this is our hypervisor, a pair of operating systems and to keep this simple, a pair of applications.

      Now, only to imagine that these applications, the operating systems and the hypervisor are all using encryption in some way.

      That might be encryption at rest or in transit, it might be public key infrastructure or it might be simple SSL or TLS.

      Whatever the requirement, it means you're going to have keys stored in many places, keys inside the applications, controlled by the operating system or held by the hypervisor.

      And all this means that keys will be handled by the CPU, held in memory and stored on storage.

      And over time, if you care about disaster recovery, you're going to have keys stored on various backups, some of which might go offsite for storage.

      Using encryption means using keys and these keys will be stored or held in various places.

      You might think this is controllable, but over time they will leave your premises and because of this, your directs control, meaning it becomes easier for these to fall into the wrong hands and become exploited.

      Now, that's where HSMs add value.

      So let's take a look.

      Now, this is a similar architecture, the same hypervisor, the same set of operating systems and applications and the same backup infrastructure.

      Only now, we've chosen to utilize a HSM.

      A HSM or hardware security module, as the name suggests, is a separate device or cluster of devices.

      It's isolated from your main infrastructure and it's inside this device that your keys are stored.

      They never leave this device.

      They're managed by this device, often generated and deleted by this device.

      Anytime you want to perform cryptographic operations, you send those to the HSM together with the data.

      The HSM performs cryptographic operations and sends the result back.

      It means you keep the same application and virtualization architecture, but instead of having to generate, to manage, store and secure keys and risk those leaking, with HSM, the keys are securely held on device.

      So that's HSMs at the high level.

      Let's finish by exploring the architecture in more detail.

      So now we have our HSM in the middle.

      Think of this as 100% separated from the rest of our infrastructure, accessible only in a highly defined way.

      Keys are created on the HSM, stored on the HSM, operations happen on the HSM and keys generally never leave the device.

      By utilizing HSMs, you create a secure island within your infrastructure, where all cryptographic operations are controlled from.

      The authentication takes place inside the device.

      This means you have an isolated security blast radius.

      Even if your corporate identity store is exploited, the identities used within the HSM are internally defined, and so can withstand this type of exploit.

      HSMs are tamper-proof and they're hardened against physical and logical attacks.

      The device is used secure on-glaves internally, which makes it almost impossible to gain access to the internal key material through direct physical means.

      Many smartphones today come with a similar cut-down version of this.

      It stores your biometric information to keep it isolated from any badly behaving software on your smartphone.

      Access to cryptographic operations within the HSM is tightly controlled.

      You need access, but assuming you have those access permissions, this access is still via industry-standard APIs, such as PKCS11, JCE and CryptoNG.

      Nothing is directly accessible.

      It's all controlled via APIs.

      Now, there's even role separation for admins, so you can define people who can admin the HSM for things like software updates, key generation and other admin tasks, but those people might not be able to perform cryptographic operations.

      Many HSMs are audited to some very stringent standards, such as those required for US government use, and it's this auditability, this access control, which makes them such a powerful type of device.

      Examples of the types of situations where you might use a HSM, and this is a very small subset there are many others, but you might use it to off-load processing for SSL or TLS onto the HSM.

      So if you have a fleet of web servers, you might have the HSM device perform heavy lifting on your behalf instead of the web servers.

      HSMs often handle this in hardware using acceleration, so you gain the benefits of secure key management and the performance boost via off-loading.

      You might also use HSMs for signing certificates for a private PKI infrastructure that you have within your business.

      This just provides a way that you can securely manage the key material used to sign your certificates.

      Now, I wanted to keep this video brief and just provide a very high-level introduction, because I'm going to be making many more videos in this series.

      In order for those to make sense, you need to understand why HSMs are needed, what they do, and how they work at a high level.

      And so that's what I covered in this video.

      Now, I hope you've enjoyed it, but that's everything for now.

      So go ahead and complete the video, and when you're ready, I look forward to you joining me in the next.

    1. Welcome to this video where I want to talk about an encryption fundamental topic, envelope encryption.

      Now this process is where you encrypt something with a key and then you encrypt that with another key.

      Now to understand why we do this and what benefits it provides we need to look visually at the process.

      So let's begin with that.

      Let's look at a typical example of how envelope encryption might be implemented within AWS and one of its services, S3 or the simple storage service.

      And we want to encrypt a huge number of private captaches.

      Now we also want to follow the best practices and ensure that each of the captaches is encrypted using a different key.

      That way if one key is leaked it only impacts that one single image.

      This is an ideal use case for KMS.

      KMS manages KMS keys which in a general sense are called key encryption keys because they're only used to encrypt data less than 4kb in size which generally means other encryption keys.

      Now KMS has its own permissions.

      You can't do anything with KMS unless you also have specific permissions to interact with KMS and any key encryption keys so KMS keys managed by the KMS product they stay inside KMS.

      You can't access them directly.

      KMS performs actions on data that you provide to it using those keys as long as you have permission to do so.

      Now there's another type of key called a data encryption key or a deck.

      Decks are created by KMS but not managed by KMS.

      That's the responsibility of the system or person generating them.

      Typically when using KMS you generate a deck using a KMS key and when that happens you get two copies of the deck returned.

      A plain text version of that deck which can be used immediately to encrypt or decrypt data and an encrypted version also known as a wrapped version.

      This encrypted version is encrypted using the KMS key which was picked when creating it.

      That's to say that the wrapped data encryption key can only be encrypted using the KMS key known as a kek which was used when KMS created that data encryption key.

      The next step in this process is the service S3 in this case which requested the data encryption key.

      It uses the plain text version to encrypt the private data and then it discards the plain text deck.

      The encrypted also known as the wrapped deck is stored alongside with the encrypted data.

      Now this means we have a unique data encryption key per object that we encrypt and an encrypted version of that data encryption key is always stored alongside the object.

      So we always have the deck available because it's stored with the object and we know that the deck is ever leaked.

      One it's encrypted so it's useless without the ability to decrypt it and also if someone did manage to decrypt the data encryption key it would only be usable to decrypt that one single encrypted object.

      Data encryption keys are almost always symmetric keys.

      Why?

      Because it's far faster to encrypt data with a symmetric key.

      The key encryption keys in a generic sense can be asymmetric or symmetric with AWS which uses KMS.

      The key encryption keys also known as KMS keys are symmetric.

      The key thing to understand point in tendon is how using a wrap key architecture allows us to manage keys at scale.

      KMS service only has in this architecture one key encryption key to manage that one kek could be used to generate millions of data encryption keys.

      Each of them used to encrypt a single object.

      Each encrypted object would be encrypted using a unique data encryption key.

      What it also means is that the admin of the service can manage the storage of the object and the keys but can't decrypt either without permissions on the key encryption key.

      Remember this uses KMS which has its own isolated permissions and the keys managed by KMS never leave KMS.

      So you have to go via the product and you're subject to its permissions.

      So this point let's have a look at the decryption flow.

      We start this decryption process with KMS and the key encryption key that it manages.

      We also have the ciphertext so the encrypted cat image that the service managers S3 and its example.

      And we also have the wrapped or encrypted data encryption key or deck which is stored along with the encrypted object.

      Now first to decrypt the encrypted object we need a plain text version of the data encryption key.

      So a request is made to KMS to unwrap this deck meaning to decrypt it.

      And since KMS doesn't manage the data encryption keys this key has to be passed to KMS as part of the request.

      Now this is where the separate permissions of KMS are important because whatever the permissions of the service or entity wanting to decrypt the data the entity or service needs permissions to decrypt the deck.

      If they're allowed KMS returns the decrypted deck and this key is then used to decrypt the ciphertext, the encrypted object giving us our cat image and then the data encryption key is discarded.

      Now a benefit of this decryption approach aside from the permission separation is that again using symmetric keys is fast.

      And also because we're only using KMS to decrypt the data encryption keys we only send small amounts of data to KMS and get small amounts of data back.

      The actual data, the large objects are never passed around.

      OK let's finish up with a few summary points.

      First asymmetric keys are flexible.

      The public part of an asymmetric key pair is public and can be widely distributed and this can be used to encrypt data that can only be decrypted using the private part of an asymmetric key.

      But they're also slow.

      It's fairly computationally heavy to encrypt using asymmetric keys.

      To contrast this symmetric keys are fast but they're difficult to securely move because the same key is used to encrypt and decrypt.

      These are not the type of thing that you can publish or otherwise exchange over non-encrypted mediums and then you end up in a catch-22 situation where you need to encrypt that key and then you have the same problem.

      What are you going to use to exchange that key?

      Envelope encryption can be the best of both if you use asymmetric key encryption keys.

      Symmetric keys are generally used to encrypt or decrypt things when speed is a priority and generally these can be secured with asymmetric keys for flexibility.

      When using KMS and AWS though, the key encryption keys are also symmetric but they're locked inside the product.

      In either case by using KMS or another key management system together with envelope encryption you have less data to send to or receive from the key storage service.

      And in addition to this, you need data encryption keys can be used per object which limits the blast radius if any of these keys are ever leaked or otherwise exposed.

      At this point I'll let you see everything I wanted to cover in this envelope encryption fundamentals lesson.

      Thanks for watching.

      At this point I'll go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

      Thank you.

    1. Welcome to this lesson where I want to provide a quick foundation into encryption.

      Now I want to keep foundation lessons as short as possible so let's jump in and get started.

      Before we get started though, I just want to cover the topics that we're going to go through in this lesson.

      I'll be starting by talking about the different approaches to encryption, so encryption at rest and encryption in transit.

      I'll follow up by talking about the different concepts, so the different components and how those fit together.

      I'll cover symmetric encryption, asymmetric encryption, including the differences between those two, and I'll finish up the lesson talking about signing and then steganography.

      Now I'll get started by talking about the different approaches to encryption, so we'll do that first.

      There are two main approaches to encryption that you will see used within AWS and the real world in general.

      Each of these is aimed at solving a very different problem.

      First we've got encryption at rest and second encryption in transit.

      Encryption at rest is designed to protect against physical theft and physical tampering, and a common example of this is a user with an encrypted laptop.

      So Nat's using her laptop as she would with any other device, but her laptop is busy encrypting or scrambling any data that it writes to the internal storage, and then decrypting that data when it reads it from the same storage into memory.

      Now there's a special piece of data that's used to encrypt and decrypt that data, and it's only known to Nat.

      Now the proper word for this is secret.

      Now with laptop encryption, this is either the password for the user logging into the laptop, or a piece of data that's derived from that, but in other types of encryption, it's more complex than that.

      What this means though, is that if Nat's laptop is stolen or tampered with, the data is encrypted at rest without the information required to decrypt it.

      It's useless to an attacker.

      If somebody steals a laptop without the passcode that Nat uses, all they have is a laptop with encrypted or scrambled data, which is useless.

      Encryption at rest is also used fairly commonly within cloud environments.

      Your data is stored on shared hardware, and it's done so in an encrypted form.

      Even if somebody else could find and access the base storage device that you were using, they couldn't access your data.

      Encryption at rest is generally used where only one party is in this case, Nat, and that party is the only person who knows the encryption and encryption team.

      The other approach to encryption is known as encryption in transit, and this is aimed at protecting data while it's being transferred between two places.

      So when Nat is using her encryption data, the data is encrypted before it exits Nat's laptop, and decrypted by the bank when it arrives, and the same process is followed in reverse.

      So the bank encrypts any data that's destined for Nat's laptop, and Nat's laptop performs the decryption process.

      What you're essentially doing with encryption in transit is to apply an encryption wrapper, a tunnel, around the raw data, and anyone looking from the outside would just see a stream of scrambled data.

      Encryption in transit is generally used when multiple individuals or systems are involved.

      So let's move on and talk about encryption concepts.

      In this part of the lesson, I want to introduce some encryption terms.

      Not all of these are immediately intuitive, and so if you haven't heard of these before, I want to confirm your understanding because I'll be using them throughout the course.

      Now we'll start with plaintext, and this is a horrible term to use for this thing, because the name gives you the impression that it's text data, and it isn't always.

      Plaintext is unencrypted data.

      It can be text, but it doesn't have to be.

      It can also be images or even applications.

      Plaintext is data that you can load into an application and use, or you can load and immediately read that data.

      The next term is an algorithm, and an algorithm is a piece of code, or more specifically a piece of maths which takes plaintext and an encryption key, which I'll talk about shortly, and it generates encrypted data.

      Now common examples of algorithms are Blowfish, AES, RC4, DES, RC5, and RC6.

      When an algorithm is being used, it needs the plaintext, and it needs a key.

      And a key is the next term I want to talk about.

      A key at its simplest is a password, but it can be much more complex.

      When an algorithm takes plaintext and a key, the output that it generates is ciphertext.

      Now just like plaintext, ciphertext isn't always text data.

      Ciphertext is just encrypted data.

      So the relationship between all these things is that encryption, it takes plaintext, it uses an algorithm and a key, and it uses those things to create ciphertext.

      Decryption is just the reverse.

      It takes ciphertext, it takes a key, and it generates plaintext.

      Now this is not all that complex at a high level, but like most things in tech, there are some details which you need to understand.

      First I want to focus on the encryption key for the next part of the lesson.

      The type of key influences how encryption is used.

      So let's look at the different types of keys and different types of encryption.

      The first type of encryption key that I want to talk about is a symmetric key.

      Symmetric keys are used as part of a symmetric encryption process.

      Now it's far easier to show you an example visually rather than just explain it.

      So here goes.

      Now as everybody knows at this point I'm a fan of animals, specifically cats.

      What you might not know is I'm also a fan of robots.

      And everybody knows that cats want to achieve world domination, and robots are working towards the robot apocalypse.

      In this example, they've allied.

      They created a plan for world domination.

      So on the left we've got the cat supreme ruler, and on the right we've got the robot general.

      Both leaders want to exchange data, their battle plans, and they want to do that without humans being able to read them in a plaintext form.

      They need to ensure that the battle plans are only ever exchanged using ciphertext, so the humans never see the plaintext battle plans.

      So step one is they agree on an algorithm to use, in this case AES 256.

      And they set to work preparing to send the plaintext battle plans.

      Now the cat ruler, because he's the party sending the data, he needs to generate a symmetric encryption key, so he needs to create that and keep it safe.

      A symmetric encryption algorithm is used, and this accepts the key and the plaintext battle plans.

      And once it's accepted both of those, it performs encryption and it outputs ciphertext, the encrypted battle plans.

      The encrypted battle plans are now secure, because they're ciphertext and nobody can decipher them without the key.

      They can be sent over any transmission method, even an insecure way to the robot general.

      The encryption removes the risk of transmitting this data in the open, so even if we handed the ciphertext over to an untrustable party and asked for him to deliver that to the robot general, that would still be safe because the ciphertext is un-desyferable without the key.

      But this is where the problem starts for our rulers.

      The robot general doesn't have the key which was used to encrypt the data.

      With symmetric encryption, the same key is used for both the encryption and the encryption processors.

      So we need to find a way to get the robot general a copy of the key that was used to encrypt the data.

      So how do we do that?

      Well, we could transfer it electronically, but that's a bad idea because if the humans get the key, it's all over.

      They can also decrypt the data.

      We could arrange an in-person meetup, but for anything which is really sensitive, this is less than ideal because the people meeting to exchange the key could be intercepted on their way.

      We could encrypt the encryption key and then transfer that key.

      Now, that would be safe because the encryption key would be protected, but we'd still need to find a safe way of transferring the key that was used to encrypt the encryption key, and that gets really complex really quickly.

      This is why symmetric encryption is great for things like local file encryption or disk encryption or lac-box, but not so useful for situations where the data needs to be transferred between two remote parties, because arranging the transit of the key is the problem, and generally we need to do that in advance so there is no delay in decrypting the data.

      If the data that we're transferring is time-sensitive, the transit of the encryption key needs to happen in advance, and that's the most complex part of this method of encryption.

      Now, if we did have a way to transfer the key securely, then the same algorithm would decrypt the data using the key and the ciphertext, and then we'd return the original plaintext battle plans.

      But there's another way of doing it, and that's to use asymmetric encryption, and this addresses some of the problems that our rulers are facing.

      It makes it much easier to exchange keys because the keys used in asymmetric encryption are themselves asymmetric.

      Now, let's look at exactly what this means.

      To use asymmetric encryption, the first stage is for the cap ruler and the robot channel to agree an asymmetric algorithm to use, and then create encryption keys for the algorithm, which logically enough will be asymmetric encryption keys.

      Asymmetric encryption keys are formed of two parts, a public key and a private key.

      For both sides to be able to send and receive to each other, then both sides would need to make both public and private keys.

      To keep the diagram simple, we're going to use the example of where the cap ruler will be sending the battle plans to the robot channel, so only the robot channel in this scenario will need to generate any keys.

      Now, a public key can be used to generate ciphertext, which can only be decrypted by the corresponding private key.

      The public key cannot decrypt data that it was used to encrypt, only the private key can decrypt that data.

      This means the private key needs to be guarded really carefully because it's what's used to decrypt data.

      If it leaks, the battle plans could be compromised.

      The public key, it's just used to encrypt, and so the robot general uploads his public key to his cloud storage so that anyone can access it.

      The worst thing that could happen to anyone who obtains the robot general's public key is that he or she could use it to encrypt plaintext into ciphertext that only the robot general could decrypt.

      So there's no downside to anyone getting hold of the robot general's public key.

      So with asymmetric encryption, there's no requirement to exchange keys in advance.

      As long as the robot general uploaded his public key to somewhere that was accessible to the world, then the first step would be for the cap ruler to download the robot general's public key.

      Remember, this isn't sensitive.

      Anyone can use it to encrypt data for the robot general, and that's it.

      That's the only thing that the public key can do in this scenario.

      So using the general's public key and the plaintext battle plans, the asymmetric algorithm would generate some ciphertext.

      The ciphertext can then be transmitted to the robot general, and once received, only the robot general could decrypt that data.

      This time, though, there's no key exchange required because the rulers are using asymmetric encryption.

      The general already has his private key, and so he provides that private key and the ciphertext to the algorithm, which decrypts the ciphertext back into plaintext, and then the robot general has a copy of plaintext battle plans.

      Asymmetric encryption is generally used where two or more parties are involved, and generally when those parties have never physically met before.

      Issues by PTP, popular email and file encryption system.

      Issues by SSL or TLS, which is a system for encrypting browser communications.

      And issues by SSH, a popular method to securely access servers using key-based authentication.

      Now, asymmetric encryption is computationally much more difficult to do than symmetric, and so many processors use asymmetric encryption to initially agree and communicate symmetric key, and then the symmetric key is used for communication between those two parties from that point onward.

      Okay, so this is the end of part one of this lesson.

      It was getting a little bit on the long side, and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part two will be continuing immediately from the end of part one.

      So go ahead, complete video, and when you're ready, join me in part two.

    1. Welcome back.

      In this video, I want to talk in general about application layer firewalls, also known as layer 7 firewalls, named after the layer of the OSI model that they operate at.

      Now I want to keep this video pretty generic and talk about how AWS implement this within their product set in a separate video.

      So let's jump in and get started.

      Now before I talk about the high level architecture and features of layer 7 firewalls, let's quickly refresh our knowledge of layer 3, 4 and 5.

      So we start with a layer 3 and 4 firewall, which is helping to secure the Categorum application.

      Now this is accessed by millions of people globally because it's that amazing.

      Now because this is layer 3 and 4, the firewall sees packets and segments, IP addresses and ports.

      It sees two flows of communications, requests from the laptop to the server, and then responses from the server back to the laptop.

      Because this firewall is limited to layer 3 and 4 only, these are viewed as separate and unrelated.

      You need to think of these as different streams of data, request and response, even though they're part of the same communication from a human perspective.

      Now if we enhance the firewall, this time adding session capability, then the same communication between the laptop and server can be viewed as one.

      The firewall understands that the request and the response are part of the same session, and this small difference both reduces the admin overhead, so one rule instead of two, but this also lets you implement more contextual security, where you can think of response traffic in the context that it's response to an original request, and treat that differently than traffic in the same direction, which is not a response.

      Now this next point is really important.

      In both cases, these firewalls don't understand anything above the layer at which they operate.

      The top firewall operates layer 3 and 4, so it understands layers 1, 2, 3 and 4.

      The bottom firewall does this, plus layer 5.

      Now what this means is that both of them can see IP addresses, ports, flags, and the bottom one can do all this, and additionally, it can understand sessions.

      Neither of them though can understand the data which flows over the top of this.

      They have no visibility into layer 7, for example, HTTP.

      So they can't see headers or any of the other data that's been transferred over HTTP.

      To them, the layer 7 stuff is opaque.

      A cat image is the same as a dog image, is the same as some malware, and this is a significant limitation, and it exposes the things that we're protecting to a wide range of attacks.

      Now layer 7 firewalls fix many of these limitations, so let's take a look at how.

      Let's consider the same architecture where we have a client on the left, and then a server or application on the right that we're trying to protect.

      In the middle we have a layer 7 firewall, and so that you'll remember it's a layer 7 firewall, let's add a robot, a smarter robot.

      With this firewall, we still have the same flow of packets and segments, and a layer 7 firewall can understand all of the lower layers, but it adds additional capabilities.

      Let's consider this example where the Categorum application is connected using a HTTPS connection.

      So encrypted HTTP and HTTP is the layer 7 protocol.

      The first important thing to realize is that layer 7 firewalls understand various layer 7 protocols, and the example we're stepping through is HTTP.

      So they understand how that protocol transfers data, its architecture, headers, data, hosts, all the things which happen at layer 7 or below.

      It also means that it can identify normal or abnormal elements of a layer 7 connection, which means it can protect against various protocol specific attacks or weaknesses.

      In this example, so a HTTPS connection to the Categorum server, the HTTPS connection would be terminated on the layer 7 firewall.

      So while the client thinks that it's connecting to the server, the HTTPS tunnel would be stripped away, leaving just HTTP, which it could analyze as it transits through the firewall.

      So a new HTTPS connection would be created between the layer 7 firewall and the back end server.

      So from the server and client's perspective, this process is occurring transparently.

      The crucial part of this is that between the original HTTPS connection and the new HTTPS connection, the layer 7 firewall sees an un-imcripted HTTP connection.

      So this is plain text, and because the firewall understands the layer 7 protocol, it can see and understand everything about this protocol stream.

      Data at layer 7 can be inspected, blocked, replaced, or tagged, and this might be protecting against adult content, spam, off-topic content, or even malware.

      So in this example, you might be looking to protect the integrity of the Categorum application.

      You'll logically allow cat pictures, but might be less okay with doggoes.

      You might draw a line and not allow other animals.

      Sheik, for example, might be considered spam.

      Maybe you're pretty open and inclusive and only block truly dangerous content such as malware and other exploits.

      Because you can see and understand one or more application protocols, you can be very granular in how you allow or block content.

      You can even replace content.

      So if adult images flow through, these can be replaced with a nice kitten picture or other baby animals.

      You can even block specific applications such as Facebook and even block the flow of business data leaving the organization onto services such as Dropbox.

      The key thing to understand is that the layer 7 firewall keeps all of the layer 3, 4, and 5 features, but can react to the layer 7 elements.

      This includes things like DNS names which are used, the rates of flow for county connections per second, you can even react to content or headers, whatever elements are contained in that specific layer 7 protocol which the firewall understands.

      Now some layer 7 firewalls only understand HTTP, some understand SMTP which is the protocol used for email delivery.

      The limit is only based on what the firewall software supports.

      Now that's everything that I wanted to cover at a high level.

      Coming up in future videos, I'm going to be covering how AWS implements layer 7 firewall capability into its product set.

      For now though, this high level understanding is what I wanted to help with in this video.

      So go ahead and complete the video.

      Thanks for watching.

      I'm already, I look forward to you joining me in the next.

    1. Welcome back and in this lesson I want to talk about Jumbo Frames as well as how they're supported within AWS.

      Now let's jump in straight away and get started.

      So what is a Jumbo Frame?

      Well the maximum Ethernet V2 frame size is 1500 bytes.

      That's what you should think about when you think of a standard frame.

      Anything bigger than this is classified as a Jumbo Frame.

      But generally when most people refer to Jumbo Frames they mean a frame with a maximum size of 9000 bytes.

      Now let's visually explore what Jumbo Frames are and how they help improve performance in certain networking situations.

      Imagine the situation where you have four EC2 instances, A, B, C and D.

      Now A and B are connected and C and D are connected.

      A and B are using standard frames to communicate and C and D are using Jumbo Frames.

      All frames have a static part which is known as the frame overhead.

      Regardless of the size of the frame this is largely a standard size.

      And then there's the frame payload and this varies in size based on the data that the frame is carrying up to a maximum size.

      For normal frames this is up to 1500 bytes.

      For Jumbo Frames this maximum increases to 9000 bytes in size.

      So this is what communications between incidents A and B might look like and remember they're using standard frames.

      Well data is split between frames and the frames are transmitted on the shared medium and there are two things that you should consider.

      First there's always this frame overhead regardless of the payload size.

      And second there's always space between frames.

      Time when nothing is being transmitted.

      This is used as one method of demarcation between frames.

      So with normal frames you have a high ratio of overhead to payload size and this is pretty inefficient.

      You also have more frames because each frame can carry up to 1500 bytes.

      This means more frames per amount of data and this means more space between frames.

      And logically this means more wasted time on the medium.

      Now compare this to how the same transfer might look with Jumbo Frames.

      See the difference?

      You have more payload data per frame.

      This means that the frame overhead represents a much smaller component of the overall frame and this is more efficient.

      Also larger frames mean more data per frame.

      This means less frames for a given amount of data.

      Less frames means you have less waste of the medium so the space between those frames and this is more efficient.

      So that's why Jumbo Frames are good.

      They're much more efficient when you're dealing with demanding network applications.

      But as always there are some considerations that you need to be aware of.

      First since Ethernet which is layer 2 generally carries higher level data inside the frames.

      If you're communicating between two devices using IP for example then you need to make sure that every step supports the same size of Jumbo Frames.

      Because if you don't you could have fragmentation.

      Also not everything within AWS supports Jumbo Frames.

      Which means that if you have a communication path where you do choose to use Jumbo Frames and part of it isn't compatible with Jumbo Frames then you will get fragmentation and that will cause you problems.

      So these are all things to keep in mind if you do decide to use Jumbo Frames.

      Now let's quickly step through some areas of AWS which do and don't support Jumbo Frames before we finish this lesson.

      So first any traffic which is outside of a single VPC does not support Jumbo Frames.

      Traffic over an inter-region VPC peering connection doesn't support Jumbo Frames.

      Same region VPC peering is compatible so that's important to understand.

      Traffic over a VPN does not support Jumbo Frames.

      Traffic using an internet gateway does not support Jumbo Frames.

      You are able to use Jumbo Frames over a direct connect and transit gateway can support Frames which are larger than usual but only up to 8,500 bytes.

      So that's everything I wanted to cover in this lesson.

      With that being said go ahead and complete this video and when you're ready I look forward to you joining me in the next.

    1. Welcome back and in this lesson I want to talk in a little bit of detail about fiber optic cables.

      If you're involved with networking in any way, then you need to be comfortable with how they work, their characteristics and the differences between the various types.

      Now this matters for the real world and if you need to work with any physical networking services, including AWS Direct Connect.

      Now let's just jump in and get started.

      Fiber optic cables are an alternative way to transmit data versus copper cables.

      Where copper cables use changes in electrical signals to transmit data over a copper medium, fiber optic cables use a thin glass or plastic core surrounded by various protective layers.

      The core is about the diameter of a human hair.

      The cable that you can see and touch is that core surrounded by a lot of protection.

      If you just handle the core on its own, it would be pretty susceptible to damage.

      Now fiber optic cables, as the name suggests, transmit light over the glass or plastic medium, so light over glass or plastic versus electrical signals over copper.

      These unique elements mean that the cable can cover much larger distances and achieve much higher speeds versus copper.

      At the time of creating this lesson, this can be in the regions of terabits per second.

      Now fiber is also resistant to electromagnetic interference known as EMI and it's less prone to being impacted by water ingress into a space where the cables are being used.

      In general, fiber offers a more consistent experience versus copper cable and so in modern networks, specifically those which require higher speeds and or larger distances, fiber is often preferred versus copper.

      You're going to see over time fiber will gradually overtake copper in almost all wired networking situations as it becomes cheaper and easier to install.

      It's already used for many global networking, metro networking and even many local area networking applications.

      So that's important to understand.

      It's going to be used more and more in the future.

      Now in terms of physical makeup, this is what a typical fiber cable looks like externally.

      There are two different things that you need to think about and this is common with any form of networking cable or any form of cable in general.

      There's the cable choice which will influence the physical characteristics so how fast data can be transferred and over what distances.

      Then you have the cable connectors and these generally affect what the cable can be connected to so linked to physical ports on the networking equipment but they can also influence some of the physical characteristics in terms of distance ability and speeds.

      Now I'm not going to detail all the different fiber cable types in this lesson.

      Instead I've included a link attached to this lesson which gives you a good overview of the common cable and connector types within the industry.

      Now I want to spend a few minutes talking about the physical construction of fiber cables.

      I've talked about how the connectors are different but it's important that you understand the different physical makeups of the cable itself.

      Now when we're talking about fiber cable, you'll see it referred to using an X/Y notation.

      For example, 9/125.

      This defines two parts of the cable.

      The first part is the diameter of the core in microns and the second part is the diameter of the cladding.

      Now the first bit that surrounds the core.

      Both of these are in microns and there are a thousand microns in a millimeter.

      Now let's talk about the different components of a fiber cable and I'm mainly going to be covering the fiber core, the fiber cladding and the buffer.

      And don't worry, the functions of each of these will make sense in a second.

      Now we're going to start with the core and this is the part of the cable where the light is carried which allows for the transfer of data.

      This part on a 9/125 cable is tinier.

      It's only 9 microns across.

      So if you look at the right of the screen, you have the core, then you have the transmitter receive optics at each side and the light flows through the core along the length of the cable.

      I'll talk more about this in a moment but the light doesn't flow in a straight line.

      Now we're bouncing off the inside edges of the core which is why the size of the core matters.

      Now surrounding the core is the cladding and this is a material which has a lower refractive index versus the core.

      And this means that it acts as a container to keep the light bouncing around inside the core.

      Different types of fiber have different sizes of core and cladding and both of them radically impact the physical characteristics of the cable.

      And this is where we move on to the less important parts of the cable.

      The core and cladding were directly responsible for the physical transmission of data but now we're moving on to the protective balance.

      So next we have the buffer and the buffer is the thing which adds strength to the cable.

      The core and cladding are generally really good at helping to carry data but really bad at withstanding any physical shocks.

      The buffer is a combination of coating and strengthening materials such as fibers made out of other materials.

      Now don't confuse this type of fiber with fiber optic.

      This is just a length of material which is designed to absorb shocks and give physical protection.

      And this buffer is surrounded by the cable jacket which is the physical thing that you see when looking at the fiber cable.

      It generally has a defined color such as green, orange or blue and this generally gives some indication on the overall capabilities of the cable.

      Now I've included a link attached to this lesson which details the common colors and what they mean in terms of the fiber optic cable capabilities.

      Now one more thing that I want to cover before we finish this lesson and that's the difference between single mode and multi mode fiber.

      The difference might seem pretty nuanced but it's really important to understand.

      Let's start with single mode.

      Single mode generally has a really small core and it's often 8 to 9 microns in size.

      And it generally uses a yellow jacket but this isn't always the case.

      Now because of this tiny core, light generally follows a fairly single and straight path down at the core.

      There's generally very little bounce and so very little distortion.

      Single mode fiber generally uses lasers and so it's more expensive in terms of the optics versus multi mode.

      Single mode because of this lack of distortion is great for long distances and it can achieve excellent speeds over these long distances.

      Now it's not the fastest type of fiber cable but if you need a combination of high speeds and long distances then it's by far the best.

      Single mode fiber can reach kilometers and can do really high speeds at those distances.

      Generally in production usage this is 10 gig and above.

      Single mode fiber cable itself is generally cheaper than multi mode fiber but the transceivers are things which send and receive light are more expensive versus multi mode.

      But this is changing over time and this will probably mean more and more single mode usage within most business applications.

      Now multi mode cable generally has a much bigger core and often uses either an orange and aqua or other coloured jacket.

      The bigger core means that it can be used with a wider range of wavelengths of light generally at the same time.

      For simplicity think of this as different colours of light travelling down the same fiber cable so these different colours can be sent at the same time and don't interfere with each other.

      More light means more data so multi mode tends to be faster.

      But that comes with a trade off because this leads to more distortion over the light over longer distances.

      For that reason multi mode historically has been used for shorter cable runs where speed and cost effectiveness is required.

      Now multi mode generally has cheaper LED based optics rather than the more expensive laser optics used within single mode fiber.

      Multi mode fiber cable will generally use the prefix OM so OM2, OM3, OM4 and so on each improving the previous ones capabilities.

      And multi mode as I mentioned before has a larger core.

      At a high level the type of cable you decide on is determined by the distances that you need to transmit data and the speed.

      So single mode is just more sturdy, there's less distortion and it can do better speeds over higher distances.

      And as the optics prices come down I suspect more people will use single mode even for shorter distances.

      Now one final thing I want to cover before we finish up with this lesson and that's fiber optic transceivers.

      Now these are generally the things which you plug into networking equipment which allows the networking equipment to connect to fiber optic cables.

      They're known as SFP transceiver modules also known as SFP or mini gibix and this stands for single form factor pluggable.

      Now these are the things which generate and send or receives light to and from the fiber optic cable.

      So these plug into networking equipment, these have optics inside which generate the light or can detect the light and these are used to translate from data to light and from lights to data that networking equipment can use.

      Now these transceivers are either multi mode or single mode and they're optimised for a specific cable type.

      So you generally buy a transceiver that's designed to be used with a certain type of cable and the transceivers will need to be the same type on both sides or both ends of the fiber optic cable.

      Now when you're talking about the connector type and the cable you're generally going to see terms such as 1000 base LX, 10G base LR or 100G base LR4.

      And these are often specified by vendors such as AWS to give you an idea on the type of cable and the connector that you need to use to plug into their equipment.

      So in the case of AWS DirectConnect the supported types are 1000 base LX, 10G base LR and 100G base LR4.

      Now at this point that's everything I wanted to cover at a high level about fiber optic cables and transceivers and once again I've included some links attached to this lesson which go into a little bit more detail if you're interested.

      At this point that's everything I wanted to cover to go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome back.

      In this lesson, I want to cover IPsec fundamentals.

      So I want to talk about what IPsec is, why it matters, and how IPsec works at a fundamental level.

      Now we have a lot of theory to cover, so let's jump in and get started.

      At a foundational level, IPsec is a group of protocols which work together.

      Their aim is to set up secure networking tunnels across insecure networks.

      For example, connecting two secure networks or more specifically, their routers, called peers, across the public internet.

      Now you might use this if you're a business with multiple sites, spread around geographically and want to connect them together, or if you have infrastructure in AWS or another cloud platform and want to connect to that infrastructure.

      IPsec provides authentication so that only peers which are known to each other and can authenticate with each other can connect.

      And any traffic which is carried by the IPsec protocols is encrypted, which means to one look at secure data which is being carried is ciphertext.

      It can't be viewed and it can't be altered without being detected.

      Now architecturally, it looks like this.

      We have the public internet which is an insecure network full of goblins looking to steal your data.

      Over this insecure network, we create IPsec tunnels between peers.

      Now these tunnels exist as they're required.

      Within IPsec VPNs, there's the concept of interesting traffic.

      Now interesting traffic is simply traffic which matches certain rules, and these could be based on network prefixes or match more complex traffic types.

      Regardless of the rules, if data matches any of those rules, it's classified as interesting traffic, and the VPN tunnel is created to carry traffic through to its destination.

      Now if there's no interesting traffic, then tunnels are eventually torn down only to be reestablished when the system next detects interesting traffic.

      The key thing to understand is that even though those tunnels use the public internet for transit, any data within the tunnels is encrypted while transiting over that insecure network.

      It's protected.

      Now to understand the nuance of what IPsec does, we need to refresh a few key pieces of knowledge.

      In my fundamental section, I talked about the different types of encryption.

      I mentioned symmetric and asymmetric encryption.

      Now symmetric encryption is fast.

      It's generally really easy to perform on any modern CPU and it has pretty low overhead.

      But exchanging keys is a challenge.

      The same keys are used to encrypt and decrypt.

      So how can you get the key from one entity to another securely?

      Do you transmit it in advance over a different medium, or do you encrypt it?

      If so, you run into a catch-22 situation.

      How do you securely transmit the encrypted key?

      That's why asymmetric encryption is really valuable.

      Now it's slower, so we don't want to be using it all the time, but it makes exchanging keys really simple because different keys are used for encryption and decryption.

      Now a public key is used to encrypt data, and only the corresponding private key can decrypt that data.

      And this means that you can safely exchange the public key while keeping the private key private.

      So the aim of most protocols which handle the encryption of data over the internet is to start with asymmetric encryption, use this to securely exchange symmetric keys, and then use those for ongoing encryption.

      Now I mentioned that because it will help you understand exactly how IPsec VPN works.

      So let's go through it.

      IPsec has two main phases.

      If you work with VPNs, you're going to hear a lot of talk about phase one or phase two.

      It's going to make sense why these are needed by the end of this lesson, but understand that there are two phases in setting up a given VPN connection.

      The first is known as Ike phase one.

      Ike or Internet Key Exchange, as the name suggests, is a protocol for how keys are exchanged in this context within a VPN.

      There are two versions, Ike version one and Ike version two.

      Version one logically is older, version two is newer and comes with more features.

      Now you don't need to know all the detail right now, just understand that the protocol is about exchanging keys.

      Ike phase one is the slow and heavy part of the process.

      It's where you initially authenticate using a pre-shared key, so a password of sorts or a certificate.

      It's where asymmetric encryption is used to agree on, create and share symmetric keys which are used in phase two.

      The end of this phase is what's known as an Ike phase one tunnel or a security association known as an SA.

      There's lots of jargon being thrown around, and I'll be showing you how this all works visually in just a moment.

      But at the end of phase one, you have a phase one tunnel, and the heavy work of moving towards symmetric keys which can be used for encryption has been completed.

      The next step is Ike phase two which is faster and much more agile because much of the heavy lifting has been done in phase one.

      Technically, the phase one keys are used as a starting point for phase two.

      Phase two is built on top of phase one and is concerned with agreeing encryption methods and the keys used for the bulk transfer of data.

      The end result is an IPsec security association, a phase two tunnel which runs over phase one.

      Now the reason why these different phases are split up is that it's possible for phase one to be established, then a phase two tunnel created, used and then torn down when no more interesting traffic occurs, but the phase one tunnel stays.

      It means that establishing a new phase two tunnel is much faster and less work.

      It's an elegant and well designed architecture, so let's look at how this all works together visually.

      So this is Ike phase one.

      The architecture is a simple one.

      Two business sites, site one on the left with the user Bob and site two on the right with the user Julie, and in the middle, the public internet.

      The very first step of this process is that the routers, the two peers at either side of this architecture need to authenticate.

      Essentially prove their identity which is done either using certificates or pre-shared keys.

      Now it's important to understand that this isn't yet about encryption, it's about proving identity.

      Proving that both sides agree that the other side should be part of this VPN.

      No keys are exchanged, it's just about identity.

      Once the identity has been confirmed, then we move on to the next stage of Ike phase one.

      In this stage we use a process called Diffie-Hellman Key Exchange.

      Now again, I'm sorry about the jargon, but try your best to remember Diffie-Hellman known as DH.

      What happens is that each side creates a Diffie-Hellman private key.

      This key is used to decrypt data and to sign things.

      You should remember this from the encryption fundamentals lesson.

      In addition, each side uses that private key and derives a corresponding public key.

      Now the public key can be used to encrypt data that only that private key can decrypt.

      So at this point, each side has a private key as well as a corresponding public key.

      At this point, these public keys are exchanged.

      So Bob has Julie's public key and Julie has Bob's public key.

      Remember, these public keys are not sensitive and can only be used normally to encrypt data for decryption by the corresponding private key.

      The next stage of the process is actually really complicated mathematics, but at a fundamental level, each side takes its own private key and the public key at the other side and uses this to derive what's known as the Diffie-Hellman key.

      This key is the same at both sides, but it's been independently generated.

      Now again, the maths is something that's well beyond this lesson, but it's at the core of how this phase of VPN works.

      And at this point, it's used to exchange other key material and agreements.

      This part you can think of as a negotiation.

      The result is that each side again independently uses this DH key plus the exchanged key material to generate a final phase one symmetrical key.

      This key is what's used to encrypt anything passing through a phase one tunnel known as the Ike Security Association.

      Now, if that process seems slow and heavy, that's because it is.

      It's both complex and in some ways simplistically elegant at the same time, but it means that both sides have the same symmetric key without that ever having to be passed between them.

      And the phase ends with this security association in place and this can be used at phase two.

      So let's talk about that next.

      So in phase two, we have a few things.

      First, a DH key on both sides and the same phase one symmetric key also on both sides.

      And then finally, the established phase one tunnel.

      During this phase, both of the peers are wanting to agree how the VPN itself will be constructed.

      The previous phase was about allowing this exchanging keys and allowing the peers to communicate.

      This phase, so Ike phase two, is about getting the VPN up and running, being in a position to encrypt data.

      So agreeing how, when and what.

      So the first part of this is that the symmetric key is used to encrypt and decrypt agreements and pass more key material between the peers.

      The idea is that one peer is informing the other about the range of cybersuits that it supports, basically encryption methods which it can perform.

      The other peer, in this example, the right one, will then pick the best shared one.

      So the best method which it also supports and it will let the left peer know.

      And this becomes the agreed method of communication.

      Next, the DH key and the key material exchanged above is used to create a new key, a symmetrical IP set key.

      This is a key which is designed for large-scale data transfer.

      It's an efficient and secure algorithm.

      And the specific one is based on the negotiation which happened above in steps one and two at this phase.

      So it's this key which is used for the encryption and decryption of interesting traffic across the VPN tunnel.

      Across each phase one tunnel, you actually have a pair of security associations.

      One from right to left and one from left to right.

      And these are the security associations which are used to transfer the data between networks at either side of a VPN.

      Now there are actually two different types of VPN which you need to understand.

      Policy-based VPNs and route-based VPNs.

      The difference is how they match interesting traffic.

      Remember, this is the traffic which gets sent over a VPN.

      So with policy-based VPNs, there are rules created which match traffic.

      And based on this rule, traffic is sent over a pair of security associations.

      One which is used for each direction of traffic.

      It means that you can have different rules for different types of traffic.

      Something which is great for more rigorous security environments.

      Now the other type of VPN are route-based VPNs.

      And these do target matching based on prefix.

      For example, send traffic for 192.168.0.0/24 over this VPN.

      With this type of VPN, you have a single pair of security associations for each network prefix.

      This means all traffic types between those networks use the same pair of security associations.

      Now this provides less functionality which is much simpler to set up.

      To illustrate the differences between route-based and policy-based VPNs, it's probably worth looking visually at the phase one and phase two architectures.

      Let's start with a simple route-based VPN.

      The phase one tunnel is established using a phase one tunnel key.

      Now assuming that we're using a route-based VPN, then a single pair of security associations is created.

      One in each direction using a single IPsec key.

      So this means that we have a pair of security associations, essentially a single phase two tunnel, running over the phase one tunnel.

      That phase two or IPsec tunnel, which is how we talk about the pair of security associations, can be dropped when there is no more interesting traffic and recreated again on top of the same phase one tunnel when new traffic is detected.

      But the key thing to understand is that there's one phase one tunnel running one phase two tunnel based on routes.

      Running a policy-based VPN is different.

      We still have the same phase one tunnel, but over the top of this, each policy match uses an SA pair with a unique IPsec key.

      And this allows us to have for the same network different security settings for different types of traffic.

      In this example, infrastructure at the top, CCTV in the middle and financial systems at the bottom.

      So policy-based VPNs are more difficult to configure, but do provide much more flexibility when it comes to using different security settings for different types of traffic.

      Now that, at a very high level, is how VPNs functions, so the security architecture, how everything interacts with everything else.

      But for now, that's everything that I wanted to cover.

      So go ahead and complete this video, and then when you're ready, I look forward to you joining me in the next.

    1. Welcome back in this video, I want to cover the differences between stateful and stateless firewalls.

      And to do that, I need to refresh your knowledge of how TCP and IP function.

      So let's just jump in and get started.

      In the networking fundamentals videos, I talk about how TCP and IP worked together.

      You might already know this if you have networking experience in the real world, but when you make a connection using TCP, what's actually happening is that each side is sending IP packets to each other.

      These IP packets have a source and destination IP, and are carried across local networks and the public internet.

      Now TCP is a layer 4 protocol which runs on top of IP.

      It adds error correction together with the idea of ports.

      So HTTP runs on TCP port 80 and HTTPS runs on TCP port 443 and so on.

      So keep that in mind as we continue talking about the state of connections.

      So let's say that we have a user here on the left, Bob, and he's connecting to the Categoram application running on a server on the right.

      What most people imagine in this scenario is a single connection between Bob's laptop and the server.

      So Bob's connecting to TCP port 443 on the server, and in doing so, he gets information back.

      In this case, many different cat images.

      Now you know that below the surface of layer 3, this single connection is handled by exchanging packets between the source and the destination.

      Conceptually though, you can imagine that each connection, in this case, is an outgoing connection from Bob's laptop to the server.

      Each one of these is actually made up of two different parts.

      First, we've got the request part where the client requests some information from the server, in this case on cat images, and then we have the response part where that data is returned to the client.

      Now these are both parts of the same interaction between the client and server, but strictly speaking, you can think of these as two different components.

      What actually happens as part of this connection setup is this.

      First, the client picks a temporary port, and this is known as an ephemeral port.

      Now typically this port has a value between 1024 and 65535, but this range is dependent on the operating system which Bob's laptop is using.

      Then once this ephemeral port is chosen, the client initiates a connection to the server using a well-known port number.

      Now a well-known port number is a port number which is typically associated with one specific popular application or protocol.

      In this case, TCP443 is HTTPS.

      So this is the request part of the connection.

      It's a stream of data to the server.

      You're asking for something, some cat pictures or a web page.

      Next, the server responds back with the actual data.

      The server connects back to the source IP of the request part, in this case Bob's laptop, and it connects to the source port of the request part, which is the ephemeral port which Bob's laptop has chosen.

      This part is known as the response.

      So the request is from Bob's laptop using an ephemeral port to a server using a well-known port.

      The response is from the server on that well-known port, but Bob's laptop on the ephemeral port.

      Now it's these values which uniquely identify a single connection.

      So that's a source port and source IP, and a destination IP, and a destination port.

      Now I hope that this makes sense so far.

      If not, then you need to repeat this first part of the video again, because this is really important to understand.

      If it does make sense, then let's carry on.

      Now let's look at this example in a little bit more detail.

      This is the same connection that we looked at on the previous screen.

      We have Bob's laptop on the left and a Caterpillar on the right.

      Obviously the left is the client and the right is the server.

      I also introduced the correct terms on the previous screen, so request and response.

      So the first part is the client talking to the server, asking for something, and that's the request, and the second part is the server responding, and that's the response.

      But what I want to get you used to is that the directionality depends on your perspective, and let me explain what I mean.

      So in this case, the client initiates the request, and I've added the IP addresses on here for both the client and the server.

      So what this means is the packets will be sent from the client to the server, and these will be flowing from left to right.

      These packets are going to have a source IP address of 119.18.36.73, which is the IP address of the client, so Bob's laptop, and they will have a destination IP of 1.3.3.7, which is the IP address of the server.

      Now the source port will be a temporary or ephemeral port chosen by the client, and the destination port will be a well-known port.

      In this case, we're using HTTPS, so TCP port 443.

      Now if I challenge you to take a quick guess, would you say that this request is outbound or inbound?

      If you had to pick, if you had to define a firewall rule right now, would you pick inbound or outbound?

      Well, this is actually a trick question, because it's both.

      From the client perspective, this request is an outbound connection.

      So if you're adding a firewall rule on the client, you would be looking to allow or deny an outbound connection.

      From the server perspective, though, it's an inbound connection, so you have to think about perspective when you're working with firewalls.

      But then we have the response part from the server through to the client.

      This will also be a collection of packets moving from right to left.

      This time, the source IP on those packets will be 1.3.3.7, which is the IP address of the server.

      The destination IP will be 119.18.36.73, which is the IP address of the client, so Bob's Laptop.

      The source port will be TCP port 443, which is the well-known port of HTTPS, and the destination port will be the ephemeral port chosen originally by the client.

      Now again, I want you to think about the directionality of this component of the communication.

      Is it outbound or inbound?

      Well, again, it depends on perspective.

      The server sees it as an outbound connection from the server to the client, and the client sees it as an inbound connection from the server to itself.

      Now, this is really important because there are two things to think about when dealing with firewall rules.

      The first is that each connection between a client and a server has two components, the request and the response.

      So the request is from a client to a server, and the response is from a server to a client.

      The response is always the inverse direction to the request.

      But the direction of the request isn't always outbound and isn't always inbound.

      It depends on what that data is together with your perspective.

      And that's what I want to talk about a bit more on the next screen.

      Let's look at this more complex example.

      We still have Bob and his laptop and the Catergram server, but now we have a software update server on the bottom left.

      Now, the Catergram server is inside a subnet which is protected by a firewall, and specifically, this is a stateless firewall.

      A stateless firewall means that it doesn't understand the state of connections.

      What this means is that it sees the request connection from Bob's laptop to Catergram, and the response of from a Catergram to Bob's laptop as two individual parts.

      You need to think about allowing or denying them as two parts.

      You need two rules.

      In this case, one inbound rule which is the request and one outbound rule for the response.

      This is obviously more management overhead.

      Two rules needed for each thing.

      Each thing which you as a human see as one connection.

      But it gets slightly more confusing than that.

      For connections to the Catergram server, so for example, when Bob's laptop is making a request, then that request is inbound to the Catergram server.

      The response, logically enough, is outbound, sending data back to Bob's laptop, but it's possible to have the inverse.

      Consider the situation where the Catergram server is performing software updates.

      Well, in this situation, the request will be from the Catergram server to the software update server, so outbound, and the response will be from the software update server to the Catergram server, so this is inbound.

      So when you're thinking about this, start with the request.

      Is the request coming to you or going to somewhere else?

      The response will always be in the reverse direction.

      So this situation also requires two firewall rules.

      One outbound for the request and one inbound for the response.

      Now, there are two really important points I want to make about stateless firewalls.

      First, for any servers where they accept connections and where they initiate connections, and this is common with web servers which need to accept connections from clients, but where they also need to do software updates.

      In this situation, you'll have to deal with two rules for each of these, and they will need to be the inverse of each other.

      So get used to thinking that outbound rules can be both the request and the response, and inbound rules can also be the request and the response.

      It's initially confusing, but just remember, start by determining the direction of the request, and then always keep in mind that with stateless firewalls, you're going to need an inverse rule for the response.

      Now, the second important thing is that the request component is always going to be to a well-known port.

      If you're managing the firewall for the Catergram application, you'll need to allow connections to TCP 443.

      The response, though, is always from the server to a client, but this always uses a random ephemeral port, because the firewall is stateless, it has no way of knowing which specific port is used for the response, so you'll often have to allow the full range of ephemeral ports to any destination.

      This makes security engineers uneasy, which is why stateful firewalls, which I'll be talking about next, are much better.

      Just focus on these two key elements that every connection has a request and a response, and together with those, keep in mind the fact that they can both be in either direction, so a request can be inbound or outbound, and a response will always be the inverse to the directionality of the request.

      Also, you'll need to keep in mind that any rules that you create for the response will need to often allow the full range of ephemeral ports.

      That's not a problem with stateful firewalls, which I want to cover next.

      So we're going to use the same architecture.

      We've got Bob's laptop on the top left, the Catergram server on the middle right, and the Software Update server on the bottom left.

      A stateful firewall is intelligent enough to identify the response for a given request, since the ports and IPs are the same, it can link one to the other, and this means that for a specific request to Catergram from Bob's laptop to the server, the firewall automatically knows which data is the response, and the same is true for software updates.

      For a given connection to a software update server, the request, the firewall is smart enough to be able to see the response or the return data from the software update server back to the Catergram server, and this means that with a stateful firewall, you'll generally only have to allow the request or not, and the response will be allowed or not automatically.

      This significantly reduces the admin overhead and the chance for mistakes, because you just have to think in terms of the directionality and the IPs and ports of the request, and it handles everything else.

      In addition, you don't need to allow the full ephemeral port range, because the firewall can identify which port is being used, and implicitly allow it, based on it being the response to a request that you allow.

      Okay, so that's how stateful and stateful firewalls work, and I know it's been a little bit abstract, but this has been intentional, because I want you to understand how they work, and sexually, before I go into more detail with regards to how AWS implements both of these different security firewall standards.

      Now, at this point, I've finished with the abstract descriptions, so go ahead and finish this video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back in this lesson I'm going to be talking about the border gateway protocol known as BGP.

      Now BGP is a routing protocol and that means that it's a protocol which is used to control how data flows from point A through point B and C and arrives at the destination point B.

      So let's jump in and get started.

      BGP as a system is made up of lots of self-managing networks known as autonomous systems or AS.

      Now an AS could be a large network, it could be a collection of routers, but in either case they're controlled by one single entity.

      From a BGP perspective it's viewed as a black box, an abstraction away from the detail which BGP doesn't need.

      Now you might have an enterprise network with lots of routers and complex internal routing, but all BGP needs to be aware of is your network as a whole.

      So autonomous systems are black boxes which abstract away from the detail and only concern themselves with network routing in and out of your autonomous system.

      Now each autonomous system is allocated a number by IANA, the Internet Assigned Numbers Authority.

      Generally ASNs are 16 bits in length and range from 0 through to 65,535.

      Now most of that range are public ASNs which are directly allocated by IANA.

      However the range from 64,512 to 65,534 are private and can be utilized within private peering arrangements without being officially allocated.

      Now you can get 32-bit ASNs, but this lesson will focus on 16-bit ones, we're only covering the basic architecture of BGP.

      Now ASNs are autonomous system numbers are the way that BGP identifies different entities within the network, so different peers.

      It's the way that BGP can distinguish between your network or your ASN and my network.

      BGP is designed to be reliable and distributed and it operates over TCP using port 179 and so it includes error correction and flow control to ensure that all parties can communicate reliably.

      It isn't however automatic, you have to manually create a peering relationship, a BGP relationship between two different autonomous systems.

      And once done those two autonomous systems can communicate what they know about network topology.

      Now a given autonomous system will learn about networks from any of the peering relationships that it has and anything that it learns it will communicate out to any of its other peers.

      And so because of the peering relationship structure you rapidly build up a larger BGP network where each individual autonomous system is exchanging network topology information.

      And that's how the internet functions from a routing perspective.

      All of the major core networks are busy exchanging routing and topology information between each other.

      Now BGP is what's known as a path vector protocol and this means that it exchanges the best path to a destination between peers.

      It doesn't exchange every path only the best path that a given autonomous system is aware of and that path is known as an AS path and autonomous system path.

      Now BGP doesn't take into account link speed or condition, it focuses on paths.

      For example, can we get from A to D using A, B, C and D or is there a direct link between A and D?

      It's BGP's responsibility to build up this network topology map and allow the exchange between different autonomous systems.

      Now while working with AWS or integrating AWS networks with more complex hybrid architectures, you might see the terms I, BGP or E, BGP.

      Now I, BGP focuses on routing within an autonomous system and E, BGP focuses on routing between autonomous systems.

      And this lesson will focus on BGP as it relates to routing between autonomous systems because that's the type that tends to be used most often with AWS.

      Now I need to stress at this point that this lesson is not a deep dive into BGP.

      All I need you to understand at this point is the high level architecture so that you can make sense of how it's used within AWS.

      So let's look at this visually and hopefully it will make more sense.

      So I want to step through an example of a fairly common BGP style topology.

      So this is Australia, the land of crocodiles and kangaroos.

      And in this example we have three major metro areas.

      We have Brisbane on the east and this has an IP address range of 10.16.0.0/16 and the router is using the IP of 10.16.0.1 and this has an autonomous system number of 200.

      We have Adelaide on the south coast using a network range of 10.17.0.0/16 and the router is using 10.17.0.1 and this has an autonomous system number of 201.

      And then finally between the two in the middle of Australia we have Alice Springs using the network 10.18.0.0/16.

      The router uses 10.18.0.1 and the autonomous system number is 202.

      Now between Brisbane and Adelaide and between Adelaide and Alice Springs is a 1 gigabit fiberlink and then connecting Brisbane and Alice Springs is a 5 megabit satellite connection with an unlimited data cap.

      BGP at its foundation is designed to exchange network topology and it does this by exchanging paths between autonomous systems.

      So let's step through an example of how this might look using this network structure and we start at the top right with Brisbane.

      And this is how the route table for Brisbane might look at this point.

      The route table contains the destination, in this case we only have the one route and it's the local network for Brisbane.

      The next column in the route table is the next hop so what IP address is needed is the first or next hop to get to that network and 0.0.0.0 in this case means that it's locally connected and this is because it's the local network that exists in the Brisbane site.

      And then finally we have the AS path which is the autonomous system path and this shows the path of the way to get from one autonomous system to another and the I in this case means that it's the origin so it's this network.

      Now the two other locations will have a similar route table at this stage so Adelaide will have one for 10.17.0.0 and Alice Springs will have one for 10.18.0.0.0/16 and both of those will have 0.0.0.0.0 as the next hop and I for the AS path because they're all local networks.

      So each of these autonomous systems so 200, 201 and 202 can have peering relationships configured so let's assume that we've linked all three so Brisbane and Alice Springs, Alice Springs and Adelaide and then finally Adelaide and Brisbane.

      Each of those peers will exchange the best paths that they have to a destination with each other so Adelaide will send Brisbane the networks that it knows about and at this point it's only itself and what it does when it exchanges this or when it advertises this is it pre-pens its AS number onto the path.

      So Brisbane now knows that to get to the 10.17.0.0 network it needs to send the data to 10.17.0.1 and because of the AS path it knows that it goes through autonomous system 201 which is Adelaide and then it reaches the origin or I and so it knows the data only has to go through one autonomous system to reach its final destination.

      Now in addition to this Brisbane will also receive an additional path advertised from Alice Springs in this case over the satellite connection and Alice Springs propends its AS number 202 onto that path so Brisbane knows to get to the 10.18.0.0/16 network.

      The next hop is 10.18.0.1 which is the Alice Springs router and it needs to go via the 202 autonomous system number which belongs to Alice Springs.

      So at this point Brisbane knows about both of the other autonomous systems and it's able to reach both of them from a routing perspective.

      Now in addition to that Adelaide will also learn about the Brisbane autonomous system because it has a peering relationship with the Brisbane autonomous system and in addition Adelaide will also in the same way learn about the network in Alice Springs because it also has a peering relationship with the Alice Springs ASN 202.

      And then finally because Alice Springs also has BGP peering relationships between it and both of the other autonomous systems it will also learn about the Brisbane autonomous system and the Adelaide autonomous system.

      At this point all three networks are able to route traffic to the other two so if you look at the route table for Alice Springs it knows how to get to the 10.16 and 10.17 networks via the ASN of 202.1 respectively.

      All three autonomous systems can talk to both of the others and this has all been configured automatically once those BGP peering relationships were set up between each of the autonomous systems but it doesn't stop there.

      This is a ring network and so there are two ways to get to every other network clockwise and anticlockwise.

      Adelaide is aware of how to get to Alice Springs so ASN 202 because it's directly connected to that and so it will advertise this to Brisbane pre-pending its own ASN onto the AS path and so Brisbane can now reach Alice Springs via Adelaide.

      So using the 201 and then 202 AS path.

      Notice how the next hop for the route given to Brisbane is the Adelaide router so 10.17.0.1 and so if we use this route table entry the traffic would go first to Adelaide and then forward it on to Alice Springs.

      Likewise Adelaide is aware of Brisbane and so it will advertise that to Alice Springs pre-pending its own AS number onto the AS path.

      So notice how this new route on the Alice Springs route table the one for 10.16.0/16 is going via Adelaide so 10.17.0.1.

      The AS path is 201 which is Adelaide, 200 which is Brisbane and then the origin.

      Now lastly Adelaide will also learn an additional route to Alice Springs but this time via Brisbane and Brisbane would pre-pend its own ASN onto the AS path so in this case we've got the additional route at the bottom for 10.18.0.0/16 but the next hop is Brisbane 10.16.0.1 and the AS path is 200 which is Brisbane and then 202 which is Alice Springs and then we've got the origin.

      Autonomous systems advertise the shortest route that they're aware of to any other autonomous systems that they have peering relationships with.

      Now at this point we're in a situation where we actually have a fully highly available network with paths to every single network.

      If any of these three sites failed then BGP would be aware of the route to the working sites.

      Notice that the indirect routes that I've highlighted in blue at the bottom of each route table have a longer AS path.

      These are none that are preferred because it's not the shortest path to the destination.

      So Brisbane for example if it was sending traffic to Alice Springs it would use the shorter path, the direct satellite connection.

      By default BGP always uses the shorter path as the preferred one.

      Now there are situations where you want to influence which path is used to reach a given network.

      Imagine that you're the network administrator for the Alice Springs network.

      Now that autonomous system has two networking connections.

      The fiber connection coming from Adelaide and the satellite connection between it and Brisbane.

      Now ideally you want to ensure that the satellite connection is only ever used as a backup when absolutely required.

      And that's for two reasons.

      Firstly it's a slower connection.

      It only operates at 5 megabits.

      And also because it's a satellite connection it will suffer from significantly higher latencies than the fiber connection between Alice Springs and Adelaide and then Adelaide and Brisbane.

      Now because BGP doesn't take into account performance or condition the satellite connection because it's the shortest path will always be used for any communications between Alice Springs and Brisbane.

      But you are able to use a technique called AS path prepending which means that you can configure BGP at Alice Springs to make the satellite link look worse than it actually is.

      And you do this by adding additional autonomous system numbers to the path.

      You make it appear to be longer than it physically is.

      Remember BGP decides everything based on path length.

      And so by artificially lengthening the path between Alice Springs and Brisbane it means that Brisbane will learn a new route.

      The old one will be removed.

      And so the new shortest path between Brisbane and Alice Springs will be the one highlighted in blue at the bottom of the Brisbane route table.

      This one will be seen a shorter than the artificially extended one using AS path prepending.

      And so now all of the data between Brisbane and Alice Springs will go via the fiber link from Brisbane through Adelaide and finally to Alice Springs.

      BGP thinks that the path from Brisbane to Alice Springs directly over the satellite connection has three hops versus the two hops for the fiber connection via Adelaide.

      And so this one will always be preferred.

      So in summary, a BGP autonomous system advertises the shortest path to a destination that it's aware of to all of the other BGP routers that it's paired with.

      It might be aware of more paths but it only advertises the shortest one.

      And it means that all BGP networks work together to create a dynamic and ever-changing topology of all interconnected networks.

      It's how many large enterprise networks function.

      It's how the internet works.

      And it's how routes are learned and communicated when using direct connect and dynamic VPNs with an AWS.

      And that's all of the theory that I wanted to cover in this lesson.

      So go ahead, finish off this video.

      And when you're ready, I look forward to you joining me in the next.

  4. Sep 2024
    1. Welcome back in this video I want to talk about SSL and TLS.

      At a very high level they do the same thing.

      SSL stands for Secure Sockets Layer whereas TLS is Transcore Layer Security.

      TLS is just a newer and more secure version of SSL.

      Now we've got a lot to cover so let's jump in and get started.

      TLS and historically SSL provide privacy and data integrity between client and server.

      If you browse through this site to Netflix, to your bank and to almost any responsible internet business, TLS will be used for the communications between the client and the server.

      TLS performs a few main functions and while these are separate, they're usually performed together and referred to as TLS or SSL.

      First, TLS ensures privacy and it does this by ensuring communications made between a client and server are encrypted so that only the client and server have access to the unencrypted information.

      When using TLS the process starts with an asymmetric encryption architecture.

      If you've watched my encryption 101 video, you'll know that this means that a server can make its public key available to any clients so that clients can encrypt data that only that server can decrypt.

      Asymmetric encryption allows for this trustless encryption where you don't need to arrange for the transfer of keys over a different secure medium.

      As soon as possible though you should aim to move from asymmetric towards symmetric encryption and use symmetric encryption for any ongoing encryption requirements because computationally it's far easier to perform symmetric encryption.

      So part of the negotiation process which TLS performs is moving from asymmetric to symmetric encryption.

      Another function that TLS provides is identity verification.

      This is generally used so that the server that you think you're connecting to, for example Netflix.com, is in fact Netflix.com.

      TLS is actually capable of performing full two-way verification but generally for the vast majority of situations it's the client which is verifying the server and this is done using public key cryptography which I'll talk more about soon.

      Finally TLS ensures a reliable connection.

      This is a very simple way to do it.

      The client can detects against the alteration of data in transit.

      If data is altered then the protocol can detect this alteration.

      Now in order to understand TLS a little better let's have a look at the architecture visually.

      When a client initiates communications with a server and TLS is used there are three main phases to initiate secure communication.

      First Cypher suites are agreed, authentication happens and then keys are exchanged.

      These three phases start from the point that a TCP connection is active between the client and the server so this is layer four.

      And at the end of the three phases there's an encryption communication channel between a client and a server.

      This each stage is responsible for one very specific set of functions.

      The first stage focuses on Cypher suites.

      Now a Cypher suite is a set of protocols used by TLS.

      This includes a key exchange algorithm, a bulk encryption algorithm and a message authentication code algorithm or MAC.

      Now there are different algorithms and versions of algorithms for each of these and specific versions and types grouped together are known as a Cypher suite.

      So to communicate the client and server have to agree a Cypher suite to use.

      Now let's step through this visually.

      We have a client and a server and at this starting point we already have a TCP connection so TCP segments between the client and the server.

      The first step is that the client does a client hello and this contains the SSL or TLS version, a list of Cypher suites that the client supports and other things like a session ID and extensions.

      Hopefully at this point the server supports one of the Cypher suites that the client also supports.

      If not then the connection fails.

      If it does then it picks a specific one and it returns this as part of the server hello.

      Now included in this server hello is also the server certificate which includes the server's public key.

      Now this public key can be used to encrypt data which the client can send to the server which only the server can decrypt using its private key.

      But keep in mind that this is asymmetric encryption and it's really computationally heavy and we want to move away from this as soon as possible.

      Now at some point in the past the server has generated a private and public key pair and it's the public part of this which is sent back to the client.

      But and this is important part of TLS is ID validation.

      If the client just confirmed that the server it's communicating with is valid then you could exploit this.

      I could create a server which pretends to be Netflix.com without being Netflix.com and this is suboptimal.

      So it's important to understand and I'll talk more about this in a second that part of the functionality provided by TLS is to verify that the server that you're communicating with is the server that it claims to be.

      The next step of the TLS process is authentication.

      The client needs to be able to validate that the server certificate the server provides is valid, that its public key is valid and as such that the server itself is valid.

      To illustrate how this works let's rewind a little from a time perspective.

      So the server has a certificate.

      Now the certificate you can think of as a document, a piece of data which contains its public key, its DNS name and other pieces of organizational information.

      Now there's another entity involved here known as a public certificate authority or CA.

      Now there are a few of these run by independent companies and your operating system and browser trust many of these authorities and which ones is controlled by the operating system and browser vendors.

      Now at some point in the past our server and let's say this is for Categoram.io created a public and private key pair and in addition it generated a certificate signing request or CSR.

      It provided the CSR to one of the public certificate authorities and in return this CA delivered back a signed certificate.

      The CA signed a certificate which means that you can verify that the CA signed that certificate.

      If your operating system or browser trusts the certificate authority then it means your operating system or browser can verify that the CA that it trusts signed that cert.

      This means that your OS or browser trusts the certificate and now the Categoram.io server that we're using as an example has this certificate and that certificate has been provided to the client as part of the server hello in Stage 1 of the TLS negotiation.

      In Stage 2 of authentication our client which has the server certificate validates that the public certificate authority signed that certificate.

      It makes sure that it was signed by that specific CA, it makes sure that the certificate hasn't expired, it verifies that the certificate hasn't been revoked and it verifies that the DNS name that the browser is using in this case Categoram.io matches the name or the names on the certificate.

      This proves that the server ID is valid and it does this using this third party CA.

      Next the client attempts to encrypt some random data and send it to the server using the public key within the certificate and this makes sure that the server has the corresponding private key.

      This is the final stage of authentication.

      If we're at this point and everything is good, the client trusts the server, its ID has been validated and the client knows that the server can decrypt data which is being sent.

      It's at this point that we move on to the final phase which is the key exchange phase.

      This phase is where we move from asymmetric encryption to symmetric encryption.

      This means it's much easier computationally to encrypt and decrypt data at high speeds.

      We start this phase with a valid public key on the client and a matching private key on the server.

      The client generates what's known as a pre-master key and it encrypts this using the server's public key and sends it through to the server.

      The server decrypts this with its private key and so now both sides have the exact same pre-master key.

      Now based on the cipher suite that's being used, both sides now follow the same process to convert this pre-master key into what's known as a master secret.

      And because the same process is followed on the same pre-master key, both sides then have the same master secret.

      The master secret is used over the lifetime of the connection to create many session keys and it's these keys which are used to encrypt and decrypt data.

      So at this point both sides confirm the process and from this point onwards the connection between the client and server is encrypted using different session keys over time.

      So this is the process that's followed when using TLS.

      Essentially we verified the identity of the server that we're communicating with, we've negotiated an encryption method to use, we've exchanged asymmetric for symmetric encryption keys and we've initiated this secure communications channel.

      And this process happens each and every time that you communicate with the server using HTTPS.

      Now that's everything I wanted to cover within this video so go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

      Thank you.

    1. Welcome back, and in this fundamentals lesson, I want to step you through how to convert decimal numbers into binary and back again, specifically relating to IP version 4 addresses.

      Now, if this is something which is new to you, I really suggest taking this slowly and focusing on each step of the process.

      If you don't understand something, just pause, think about it, and if needed, restart the video.

      I promise, once you can do this process, it will really help you with networking.

      The process is easy once you've done it a few times, so I want to explain it as simply as I can at first, and then suggest that you get some practice.

      So let's jump in and get started.

      Before we start with the actual process, I want to set the scene as to why you need to understand it.

      When dealing with IP addresses as a human, you might see something which looks like this.

      133.33.33.7.

      This is an IP address represented in what's known as dotted decimal notation.

      Four decimal numbers ranging from 0 to 255 separated by periods.

      So that's what a human sees.

      A computer sees this, the same IP address put in binary.

      Now, this is actually a 32-bit binary number, specifically four sets of 8 bits.

      Now, 8 bits is known as a byte, each byte is also known as an octet, but crucially, it's the same number as the decimal version.

      To understand much of how the internet protocol works, so things like IP ranges, prefixes, subnet masks and routing, you have to be able to convert between the two, so decimal and binary.

      All of these things only make sense when viewing IP addresses in binary, so when it comes to networking, being able to convert between decimal and binary or vice versa really is a superpower, and that's what I want to teach you in this lesson.

      So let's move on.

      I want to start with the process of converting a decimal IP address into binary.

      This is actually the more complex direction.

      Decimal to binary is confusing at first, whereas binary to decimal is much easier.

      When you're just learning this process, I find that it's easier to tackle it in byte-sized pieces.

      Let's say that you want to convert this IP, so 133.33.33.7, into binary.

      Well, each decimal in between the dots, so 133, 33, 33 and 7 is a number between 0 to 255.

      So I find it easier, at least initially, to tackle each one of these numbers individually, working left to right.

      So that's what we're going to do in this lesson.

      I'm going to step through the maths involved to convert 133 and 33 into binary, and then I'm going to ask you to follow the same process for the last two numbers.

      Now, before I step through this process, let me introduce this table.

      This helps us with binary maths because it tells us what the decimal value is for each position in binary.

      Now, this table has eight positions, shown as 1 to 8 on the table, and so it works for 8-bit binary numbers.

      Remember, each part of an IP address, each number between the periods, is actually an 8-bit binary number.

      Now, each position in this table has a value, so 128, then 64, then 32, 16, 8, 4, 2 and finally 1.

      It means a binary 1 in that position has the associated value in decimal.

      So a 1 at position 1 is 128 in decimal, a 1 at position 4 is 16 in decimal, and a 1 at position 8 is 1 in decimal.

      We're going to use this table to help us convert decimal numbers into binary.

      We're going to move in the table from left to right, starting at position 1, moving through to position 8.

      At each position, we're going to follow a process.

      It's going to be the same process with a number of rules, and if you learn how to follow these rules, it makes the conversion really easy.

      So once again, for a given decimal part of the IP address, so in this case, the first one we're going to look at is 1, 3, 3.

      We're going to work through this table left to right, and as I mentioned earlier, I'm going to demonstrate the process for 1, 3, 3 and 33, and then you're going to follow the process for 33 and 7.

      So we start on the table in position 1, we compare our decimal numbers, so 133, to the value in the current position of the table, which is 128.

      Now, rule number one is that if our number is smaller than the corresponding binary position value, then you write 0 in that position at the table that you're in, in this case, position 1, and then you move on to the next position, starting processing using rule number one again.

      However, if our number is greater or equal to the binary position value, then rule 2 applies.

      Now, this is the case for us because 133 is more than 128.

      So what we do is we minus the binary position value, so 128, from our number, so that means 133 minus 128, this leaves us with 5.

      So 5 is the remaining decimal value that we have.

      And then we write 1 in this column and move on to position 2.

      A binary 1 in this position is equal to 128, so all that we've done is transfer 128 of the decimal value into the corresponding binary value.

      So we've added a binary 1 here and we've removed 128 from our decimal number, leaving 5.

      And now we can continue the process to convert this remaining 5 into binary.

      So now we reset the process, we move to position 2 in the table and start evaluating again using rule number 1.

      So we're comparing our remaining value, so we have 5 in decimal, against the binary position value in position 2, which is 64.

      So now we compare our number 5 to the value in the table, which is 64.

      So in our case, our decimal value of 5 is smaller than the binary value of position 2, and so we're using rule number 1.

      We add 0 into this column in the table and then we go right back to the start of the process.

      We move on to position 3 and we start evaluating against rule number 1.

      We repeat the same process for positions 3, 4 and 5.

      Our decimal value of 5 is less than all of those values, so 32, 16 and 8.

      So we add 0 in each of those columns and we move on.

      So we've evaluated all of those and they match rule number 1 and so we add 0 and we move on.

      We're just following the same basic rules.

      So now we're at position 6 in the table, we compare our remaining decimal number, so 5, against the value in position 6, which is 4.

      Now it's not smaller, so we move past rule 1, it is larger or equal and so we use rule number 2.

      We add a value of 1 in this binary position and minus this binary position value from our number, so 5 minus 4 equals 1.

      So we have 1 remaining in decimal.

      A binary value of 1 in position 6 is equal to 4 in decimal.

      So we've just added this to the binary number and removed it from the decimal number.

      What this process is doing, bit by bit, is removing value from the decimal value and adding it to the binary 1.

      First it was 128 of the decimal number and just now it was 4 of the decimal number.

      And next we move on to the next position in the table.

      So now we're looking at position 7.

      We're comparing our remaining decimal value of 1 with the binary position value.

      First we're evaluating against rule 1 and because our remaining value of 1 is less than the binary position value, we use a 0 in this column and we move on to the next position.

      So now we're on position 8.

      Again we do a comparison.

      Our decimal value is 1.

      The binary position value is 1 and so we evaluate this against rule 1.

      It's not less.

      We evaluate it against rule 2.

      It is equal or larger.

      And so we add a 1 in this column at the table and then we remove the binary position value from our decimal value.

      So 1 minus 1 is 0 and because we have 0, the process is finished.

      And this means that 133 in decimal is expressed as 1, 0, 0, 0, 1, 0, 1 in binary.

      Now this is one example, so this is converting one part of an IP address.

      Let's run this process through again with the next part of this IP address.

      So this time we're converting the second decimal part of the IP, so 33, into binary.

      We start again with our same process and our same table and we move through the table left to right.

      We start with position 1.

      We compare our decimal number, so 33, to position 1's value, which is 128.

      Is our number less than 128?

      Yes it is and so we use rule number 1.

      We add a 0 into this position at the table and then move on to position 2.

      We compare our decimal number to the value in position 2.

      Is our number less than 64?

      Yes it is and so we use rule number 1.

      We add a 0 into the table and move on to position 3.

      We compare our decimal number, so 33, to position 3's value, which is 32.

      Is our decimal number less than 32?

      No it's not, so we skip rule number 1 and move to rule number 2.

      We minus 32 from our decimal number 33, leaving us with 1's.

      So we've transferred 32 at the decimal value into the binary value and then we move on to position 4 in the table.

      Now hope at this point you start to feel more comfortable and we can speed up.

      At this point we're at position 4 and we can simply repeat the process.

      We compare values if our remaining decimal value is less than the value in the table for that position and we add 0.

      And this is the case for positions 4, 5, 6 and 7.

      So we add 0's into all of those positions.

      Then finally we have position number 8.

      And we compare our remaining decimal number, which is 1, with the value in position 8, which is also 1.

      Is our value less?

      No it's not, so rule number 1 isn't used.

      Is our value larger or equal?

      Yes it is, and so we use rule number 2.

      So we write down 1 in position number 8.

      We minus the binary position value, which is 1, from this number, leaving us 0, and that means we've finished the process.

      So the binary value for 33 is 0, 0, 1, 0, 0, 0, 0, 1.

      Now I want you to try and do this process on your own, on some paper, for the third value of 33, without looking at this lesson.

      This process is the same, but it will let you practice yourself.

      And once you've reached the same value as me, you can follow the same process for the fourth decimal value of 7.

      Once you've done them all, you'll have the full 32-digit binary number, which represents 133.33.33.7.

      So go ahead and pause the video and do the final two calculations yourself, and you can resume it once you've finished.

      Okay, so I hope that was pretty easy for you, and what you should have for the last decimal value of 7, is a binary value of 0, 0, 0, 0, 0, 1, 1, 1.

      If you did, awesome.

      That means you've completed it successfully.

      If not, don't worry, just watch the first part of this video again, and repeat the process.

      Now at this point I want to show you how this works in reverse, converting binary to decimal.

      This time let's say that you want to convert this binary IP address into decimal.

      So both of these are one and the same.

      To start with, we break the IP address into four sections, and this is the same for the binary and the dotted decimal versions.

      So each decimal part between the dots represents the corresponding 8-bit binary value.

      So I've colored these in red, blue, green, and yellow, so left, middle, left, middle, right, and right.

      So this is one and the same.

      They're the same IP address expressed in binary on the left and decimal on the right.

      Now just as before, the easiest way to tackle this is to work left to right, working on one octet, one by one.

      The process of converting this in this direction is much easier.

      We have the conversion table just as before.

      Each binary bit left to right has a value.

      One to eight on the left and one on the right, and we read them one by one.

      So we need to the 8-bit binary components, so each of the different colored squares we go through from left to right, and we look whether there's ones or zeros in each position.

      If there's a one, we take the corresponding number from the table at the bottom of the screen.

      So the first example, the one on the left represents 128.

      So we write that down.

      We write 128 and then a plus.

      If there's a zero, and this is the case for the following four bits, so zero, zero, zero, zero, then we add a zero in our equation.

      So we have one to eight, plus zero, plus zero, plus zero, plus zero.

      Then we have another one, so we look at the table at the bottom for the corresponding binary position value.

      In this case, the number four, so we add that.

      Then we have a zero, so we put zero.

      And then we have another one, so we again look for the corresponding value in the table, which is a one.

      And we add all of those together to get the result.

      In this case, 133.

      So this represents the first part of the IP address, and we follow the same process for all of the remaining parts.

      So zero, zero, one, zero, zero, zero, zero, one represents 33.

      We just take each of the ones in each of the components of the IP address and look up the corresponding binary position value in the table.

      So in the second component of the IP address, there's a one in position three, which is 32, and a one in position eight, which is one, and this represents 33.

      The same is true in the third component, and in the fourth component, there's a one in position six, seven, and eight, which represents four, two, and one.

      So we add four, two, and one together to get the result and value.

      So if you follow that process bit by bit for each eight-bit component of the binary IP address, then you will end up with the dotted decimal version of the IP address, which is exactly what we've done here.

      So why don't you go ahead and pick a random IP address and follow the same process through and see if you get the correct result.

      And then once you have the correct result, take that dotted decimal IP address and follow the previous process to convert it from decimal to binary, and you should end up with the same result that you started with.

      If you do, that means you understand the end-to-end process of binary to decimal and decimal to binary, and I promise you this does represent a superpower, so it's really important that you understand.

      At this point, I'll let everything I want to cover, so go ahead and complete the video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back and in this lesson I want to cover a few really important topics which will be super useful as you progress your general IT career, but especially so for anyone who is working with traditional or hybrid networking.

      Now I want to start by covering what a VLAN is and why you need them, then talk a little bit about Trump connections and finally cover a more advanced version of VLANs called Q in Q.

      Now I've got a lot to cover so let's just jump in and get started straight away.

      Let's start with what I've talked about in my technical fundamentals lesson so far.

      This is a physical network segment.

      It has a total of eight devices, all connected to a single, layer 2 capable device, a switch.

      Each LAN, as I talked about before, is a shared broadcast domain.

      Any frames which are addressed to all Fs will be broadcast on all ports of the switch and reach all devices.

      Now this might be fine with eight devices but it doesn't scale very well.

      Every additional device creates yet more broadcast traffic.

      Because we're using a switch, each port is a different collision domain and so by using a switch rather than a layer 1 hub we do improve performance.

      Now this local network also has three distinct groups of users.

      We've got the game testers in orange, we've got sales in blue and finance in green.

      Now ideally we want to separate the different groups of devices from one another.

      In larger businesses you might have a requirement for different segments of the network from normal devices, for servers and for other infrastructure.

      Different segments for security systems and CCTV and maybe different ones for IoT devices and IP telephony.

      Now if we only had access to physical networks this would be a challenge.

      Let's have a look at why.

      Let's say that we talk each of the three groups and split them into either different floors or even different buildings.

      On the left finance, in the middle game testers and on the right sales.

      Each of these buildings would then have its own switch and the switches in those buildings would be connected to devices also in those buildings.

      Which for now is all the finance, all the game tester and all the sales teams and machines.

      Now these switches aren't connected and because of that each one is its own broadcast domain.

      This would be how things would look in the real world if we only had access to physical networking.

      And this is fine if different groups don't need to communicate with us so we don't require cross domain communication.

      The issue right now is that none of these switches are connected so the switches have no layer 2 communications between them.

      If we wanted to do cross building or cross domain communications then we could connect the switches.

      But this creates one larger broadcast domain which moves us back to the architecture on the previous screen.

      What's perhaps more of a problem in this entirely physical networking world is what happens if a staff member changes role but not building.

      In this case moving from sales to game tester.

      In this case you need to physically run a new cable from the middle switch to the building on the right.

      If this happens often it doesn't scale very well and that is why some form of virtual local area networking is required.

      And that's why VLANs are invaluable.

      Let's have a look at how we support VLANs using layer 2 as the OSI 7-Line model.

      This is a normal Ethernet frame.

      In the context of this lesson what's important is that it has a source and destination MAC address fields together with a payload.

      Now the payload carries the data.

      The source MAC address is the MAC address of the device which is creating and sending the frame.

      The destination MAC address can contain a specific MAC address which means that it's a unique S-frame to a frame that's destined for one other device.

      Or it can contain all F's which is known as a broadcast.

      And it means that all of the devices on the same layer 2 network will see that frame.

      What a standard frame doesn't offer us is any way to isolate devices into different parts, different networks.

      And that's where a new standard comes in handy which is known as 802.1Q, also known as .1Q. .1Q changes the frame format of the standard Ethernet frame by adding a new field, a 32-bit field in the middle in Scion.

      The maximum size of the frame as a result can be larger to accommodate this new data. 12 bits of this 32-bit field can be used to store values from 0 through to 4095.

      This represents a total of 4096 values.

      This is used for the VLAN ID or VID.

      A 0 in this 12-bit value signifies no VLAN and 1 is generally used to signify the management VLAN.

      The others can be used as desired by the local network admin.

      What this means is that any .1Q frames can be a member of over 4,000 VLANs.

      And this means that you can create separate virtual LANs or VLANs in the same layer 2 physical network.

      A broadcast frame so anything that's destined to all PEPs would only reach all the devices which are in the same VLAN.

      Essentially, it creates over 4,000 different broadcast domains in the same physical network.

      You might have a VLAN for CCTV, a VLAN for servers, a VLAN for game testing, a VLAN for guests and many more.

      Anything that you can think of and can architect can be supported from a networking perspective using VLANs.

      But I want you to imagine even bigger.

      Think about a scenario where you as a business have multiple sites and each site is in a different area of the country.

      Now each site has the same set of VLANs.

      You could connect them using a dedicated wide area network and carry all of those different company specific VLANs and that would be fine.

      But what if you wanted to use a comms provider, a service provider who could provide you with this wide area network capability?

      What if the comms provider also uses VLANs to distinguish between their different clients?

      Well, you might face a situation where you use VLAN 1337 and another client of the comms provider also uses VLAN 1337.

      Now to help with this scenario, another standard comes to the rescue, 802.1AD.

      And this is known as Q in Q, also known as provider bridging or stacked VLANs.

      This adds another space in the frame for another VLAN field.

      So now instead of just the one field for 802.1Q VLANs, now you have two.

      You keep the same customer VLAN field and this is known as the C tag or customer tag.

      But you add another VLAN field called the service tag or the S tag.

      This means that the service provider can use VLANs to isolate their customer traffic while allowing each customer to also use VLANs internally.

      As the customer, you can tag frames with your VLANs and then when those frames move onto the service provider network, they can tag with the VLAN ID which represents you as a customer.

      Once the frame reaches another of your sites over the service provider network, then the S tag is removed and the frame is passed back to you as a standard .1Q frame with your customer VLAN still tagged.

      Q in Q tends to be used for larger, more complex networks and .1Q is used in smaller networks as well as cloud platforms such as AWS.

      For the remainder of this lesson, I'm going to focus on .1Q though if you're taking an advanced networking course of mine, I will be returning to the Q in Q topic in much more detail.

      For now though, let's move on and look visually at how .1Q works.

      This is a cut down version of the previous physical network I talked about, only this time instead of the three groups of devices we have two.

      So on the left we have the finance building and on the right we have game testers.

      Inside these networks we have switches and connected to these switches are two groups of machines.

      These switches have been configured to use 802.1Q and ports have been configured in a very specific way which I'm going to talk about now.

      So what makes .1Q really cool is that I've shown these different device types as separate buildings but they don't have to be.

      Different groupings of devices can operate on the same layer to switch and I'll show you how that works in a second.

      With 802.1Q ports and switches are defined as either access ports or trunk ports and access ports generally has one specific VLAN ID or vid associated with it.

      A trunk conceptually has all VLAN IDs associated with it.

      So let's say that we allocate the finance team devices to VLAN 20 and the game tester devices to VLAN 10.

      We could easily hit any other numbers, remember we have over 4,000 to choose from, but for this example let's keep it simple and keep 10 and 20.

      Now right now these buildings are separate broadcast domains because they have separate switches which are not connected and they have devices within them.

      Two laptops connected to switch number one for the finance team and two laptops connected to switch number two for the game tester team.

      Now I mentioned earlier that we have two types of switch ports in a VLAN cable network.

      The first are access ports and the ports which the orange laptops on the right are connected to are examples of access ports.

      Access ports communicate with devices using standard Ethernet which means no VLAN tags are applied to the frames.

      So in this case the laptop at the top right sends a frame to the switch and let's say that this frame is a broadcast frame.

      When the frame exits an access port it's tagged with a VLAN that the access port is assigned to.

      In this case VLAN 10 which is the orange VLAN.

      Now because this is a broadcast frame the switch now has to decide what to do with the frame and the default behaviour for switches is to forward the broadcast out of all ports except the one that it was received on.

      For switches using VLANs this is slightly different.

      First it forwards to any other access ports on the same VLAN but the tagging will be removed.

      This is important because devices connected to access ports won't always understand 802.1Q so they expect normal untagged frames.

      In addition the switch will fold frames over any trunk ports.

      A trunk port in this context is a port between two switches for example this one between switch two and switch one.

      Now a trunk port is a connection between two dot 1Q capable devices.

      It forwards all frames and it includes the VLAN tagging.

      So in this case the frame will also be forwarded over to switch one tagged as VLAN 10 which is the gain tester VLAN.

      So tagged dot 1Q frames they only get forwarded to other access ports with the same VLAN but they have the tag stripped or they get forwarded across trunk ports with the VLAN tagging intact.

      And this is how broadcast frames work.

      For unicast ones which go to a specific single MAC address well these will be either forwarded to an access port in the same VLAN that the specific device is on or if the switch isn't aware of the MAC address of that device in the same VLAN then it will do a broadcast.

      Now let's say that we have a device on the finance VLAN connected to switch two.

      And let's say that the bottom left laptop sends a broadcast frame on the finance VLAN.

      Can you see what happens to this frame now?

      Well first it will go to any other devices in the same VLAN using access ports meaning the top left laptop and in that case the VLAN tag will be removed.

      It will also be forwarded out of any trunk ports tagged with VLAN 20 so the green finance VLAN.

      It will arrive at switch two with the VLAN tag still there and then it will be forwarded to any access ports on the same VLAN so VLAN 20 on that switch but the VLAN tagging will be removed.

      Using virtual LANs in this way allows you to create multiple virtual LANs or VLANs.

      With this visual you have two different networks.

      The finance network in green so the two laptops on the left and the one at this middle bottom and then you have the gain testing network so VLAN 10 meaning the orange one on the right.

      Both of these are isolated.

      Devices cannot communicate between VLANs which are led to networks without a device operating between them such as a layer 3 router.

      Both of these virtual networks operate over the top of the physical network and it means that now we can configure this network in using virtual configuration software which can be configured on the switches.

      Now VLANs are how certain things within AWS such as public and private vifs on direct connect works so keep this lesson in mind when I'm talking about direct connect.

      A few summary points though that I do want to cover before I finish up with this lesson.

      First VLANs allow you to create separate layer 2 network segments and these provide isolation so traffic is isolated within these VLANs.

      If you don't configure and deploy a router between different VLANs then frames cannot leave that VLAN boundary so they're virtual networks and these are ideal if you want to configure different virtual networks for different customers or if you want to access different networks for example when you're using direct connect to access VPCs.

      VLANs offer separate broadcast domains and this is important.

      They create completely separate virtual network segments so any broadcast frames within a VLAN won't leave that VLAN boundary.

      If you see any mention of 802.1Q then you know that means VLANs.

      If you see any mention of VLANs stacking or provide a bridging or 802.1AD or Q in Q this means nested VLANs.

      So having a customer tag and a service tag allowing you to have VLANs in VLANs and these are really useful if you want to use VLANs on your internal business network and then use a service provider to provide wide area network connectivity who also uses VLANs and if you are doing any networking exams then you will need to understand Q in Q as well as 802.1Q.

      So with that being said that's everything I wanted to cover.

      Go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome back and in this video I want to step through the architecture and challenges of distributed denial of service attacks known as DDoS attacks.

      Now we've got a lot to cover, so let's get started.

      Distributed denial of service attacks come in many forms, a few different ways of achieving the same end goal, which is to overload websites or other internet-based services.

      The idea is to generate some kind of traffic which competes against legitimate connections and overloads the hardware or software providing the service.

      Imagine trying to get into an Apple Store on the day when a new iPhone is released.

      How hard is it to get into the store and get service?

      What if you added 100,000 random people who just want to queue for no reason and waste the time of the Apple Store staff?

      That's the physical equivalent of a DDoS attack.

      The challenge when dealing with DDoS attacks comes from the distributed nature of those attacks.

      It's hard to identify and block traffic because there can be millions of IP addresses involved with larger internet-scale attacks.

      Dealing with DDoS attacks requires specific hardware or software protections.

      We won't be covering those in this video, I'm limiting this to just discussing how DDoS attacks work, so the architecture of all the different types of DDoS attacks.

      Now DDoS attacks themselves generally fit into one of three categories.

      First, application layer attacks such as HTTP floods, and these take advantage of the imbalance of processing between client and server.

      It's easy for you to request a web page, but it's often very complex for a server to deliver that same page.

      If you multiply that load difference by a billion, it's easy for you to request a web page, but it's often very complex for a server to deliver that same page.

      If you multiply that load difference by a billion, then you can have a potentially devastating attack.

      Next, we've got protocol-based attacks such as SYNFLUDS, and SYNFLUDS takes advantage of the connection-based nature of requests.

      Normally, a connection is initiated via a three-stage handshake, which I detailed in a separate video of this series.

      While SYNFLUDS spoof a source ID address and initiate the connection attempt with a server, the server tries to perform step-to-other handshake, but it can't contact the source address because it's spoofed.

      In general, it hangs here waiting for a specified duration, and this consumes network resources.

      And again, if you multiply this effect by a billion, this can have significant impact on your ability to provide a service.

      Lastly, we have volumetric attacks such as DNS amplification, and this relies on how certain protocols such as DNS only take small amounts of data to make the request such as a DNS resolution request, but in response to that, they can deliver a large amount of data.

      So one example is an attack of this nature might make requests to DNS servers with a large number of independent requests, where the source address is spoofed to be the actual IP address of our website.

      And their servers, potentially hundreds or thousands of them, respond to what they see as legitimate requests and overwhelm a service.

      D-dot attacks are often orchestrated by one or a small number of people who are in control of huge botnets, and botnets are constructed of machines such as your laptop or your desktop infected with malware.

      Most attacks come from these botnets which are constructed from infected hosts, and the owners of these hosts don't even realize that they're part of the attack.

      Now let's look at how these attacks work visually, but before we do that, it's worth reviewing what a valid application architecture should look like.

      When a website is working as intended, it looks something like this.

      First, we have a number of servers which provide the website functionality, in this case, Categoram.io.

      These servers are normally provisioned either based on normal load plus a bit of extra as a buffer, or they're built to autoscale, which means adding more servers when loading creases and removing servers as load decreases.

      Now these servers run within a hosting environment which is connected to the public internet via a data connection, which, depending on the speed of this connection, has a limited amount of data that it can transfer and a limit on the number of connections it can handle.

      Then our application has users who are using a mobile application to upload their latest captures using TCP port 443.

      Now this is HTTPS, and these connections move across our data connection and arrive at the application servers.

      Now in normal circumstances, the vast majority of these connections will be from legitimate users of the application.

      So this is how it should work.

      We have an application, the servers are sized appropriately, we have an appropriate data connection, and our users are accessing the application using this infrastructure.

      Now let's step through what happens with the various different forms of DDoT attack.

      The first type of DDoT attack is the application layer attack.

      Architecturally, behind the scenes we have an attacker who is controlling a network of compromise machines known as a botnet.

      This botnet, or more specifically, the machines which form the botnet, are distributed geographically.

      [Pause] [Pause] In most circumstances, the real owners of these machines have no knowledge that they've been compromised.

      An application-led DDoT attack, as I mentioned at the start of this video, uses the computational imbalance of client-server communications as an attack method.

      It's easy, for instance, for the botnets to make simple requests to the application.

      In this case, a HTTP GET of a page called reallycomplex.php.

      The botnet floats hundreds of thousands, or even tens of thousands of these requests, each of them directed towards the Catergram servers.

      This would mean millions or more of these really simple requests, which are all requesting, are reallycomplex.php page.

      The issue is that while making these requests is simple, responding to these requests can be computationally expensive, and this can have disastrous effects on the servers.

      It's like throwing hand grenades.

      They're easy to throw, but they're much more difficult to deal with at the receiving end.

      The effect is that our servers, or the data connection, won't have the capacity required to deal with the requests in total.

      The fake attack-based requests will prevent the legitimate requests reaching the servers in a timely way, and this can cause performance issues or failures, essentially a general decrease in service levels.

      Now, as I mentioned earlier in this video, you can't simply block traffic from individual machines, because there can be millions of them, and the data they're sending can in many ways look exactly the same as legitimate traffic, and this is why you have to handle DDoS attacks in a very specific way.

      Now, at this point, let's move on and take a look at another type of DDoS attack.

      This time, it's a protocol-based attack.

      So, with a protocol-based attack, we follow the same basic architecture, where a single or a small group of attackers is controlling a large botnet, and this botnet is constructed of compromised hosts.

      Now, with a protocol attack such as a SYN flood, essentially what happens is a botnet generates a huge number of spoofed SYNs, and SYNs are the initial part of this three-way connection handshake.

      So, essentially, all of these individual machines attempt to initiate a connection with the Catergram.io infrastructure, but crucially, they're using a spoofed IP address.

      In normal circumstances, if these were real connections, what should happen is our server infrastructure would respond back with SYN acts, which are the second part of the three-way handshake.

      Normally, these connection attempts would be from real IP addresses, so IP addresses which are expecting to receive this second part of the three-way handshake.

      But because our botnet has initiated these connections with spoofed IP addresses, there won't be anything on the receiving end for our servers to communicate with, and so, in this case, these requests will simply be ignored.

      Because they're being ignored, it means the connections will stay in this hung state.

      The network resources which would otherwise be used for legitimate connections are waiting for this second part of the three-way handshake, and because the botnet is providing millions and millions of these fake connections, it can mean that network resources are completely consumed with these fake connections, and that means that our legitimate connections won't be able to connect into our infrastructure.

      Essentially, by generating this protocol-based attack, this SYN flood, we're preventing the network resources being used for legitimate requests, and so we're essentially significantly impacting the network capability of this application infrastructure.

      So, because this three-way handshake is designed to work with slower or less reliable connections, our Catergram.io infrastructure will wait.

      It will attempt to connect to these fake source IP addresses.

      And while these connections are waiting for the second part of the three-way handshake, these resources can't be used for legitimate connections.

      And so, if the botnet is large enough, if it contains a sufficient number of compromised hosts, then it can, in theory, completely take down the service provided by the Catergram.io infrastructure.

      Now, let's move on to the third type of attack which I want to cover in this video, and that's known as a volumetric or amplification-based attack.

      Now, this type of attack is still orchestrated by a single person or a small group of people, but with this type of attack, the size of the botnet can be much smaller, because an amplification attack exploits a protocol data imbalance.

      So, a situation where only a small amount of data is required to initiate a request, but the response to that request is much larger.

      In this case, our smaller botnet makes a large number of requests to a large number of DNS servers.

      The requests can be made to a large number of DNS servers and be done frequently, because the amount of data that it takes to make the request is relatively small.

      Now, the botnet will use a spoofed IP address, and it will use the IP address of our application infrastructure.

      So, rather than the DNS servers responding to the botnet, the DNS servers will all respond to our application servers.

      Now, the volume of data in each of those responses is much larger than the volume of data making the initial query to DNS.

      Because of this, the application servers will be quickly overwhelmed.

      This will generally affect the data connection to our application infrastructure, rather than the application server itself, and this will mean that legitimate application users experience degraded levels of performance, because they're competing to use the same total capacity of the application data connection with these fake responses coming in from all of these DNS servers.

      So, this type of attack does impact...

      Our application's ability to provide service, because the amount of data that our connection provides is consumed, but it's done so in a way which uses amplification.

      So, rather than the botnet being required to consume the same amount of bandwidth as our application needs to tolerate, this type of attack can use a tiny amount of bandwidth to initiate the attack, but consume a large amount of bandwidth on the application side, and this makes this type of attack ideally suited to take down larger websites or applications.

      So, these are three different common types of DDoS attacks, which you might come across as a solutions architect, an engineer, or a developer.

      The important thing to understand about all types of DDoS attack is they can't be combated with normal network protection.

      So, because of the distributed nature of the attacks, it's not practical to implement single IP address blocks.

      If you're going to block an entire botnet, then you need to block potentially thousands, tens of thousands, hundreds of thousands, or even millions of IP addresses.

      What's more, if you use a volumetric or amplification style attack, then the actual machines performing the attack might not even be malicious.

      In this case, if you're taking advantage of DNS servers using a DNS amplification attack, then these servers, from their perspective, are doing nothing malicious.

      They're just responding to requests.

      And so, you have to be really careful that in order to mitigate a DDoS attack, you're not actually blocking legitimate traffic or impacting your application's ability to provide a service.

      If you block all DNS servers, then potentially you can have other issues with your application.

      Now, AWS and other cloud environments do provide products and services which are specifically designed to help you combat DDoS attacks.

      And now that you're aware of the architecture and how these attacks can impact your application, it's going to be much easier for you to understand these different products and services.

      Now, with that being said, that's everything that I wanted to cover in this video.

      So, go ahead and complete the video.

      And when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back.

      This is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      Subnetting is the process of breaking networks up into smaller and smaller pieces.

      I've just talked about the class A, B, C, D, and E ranges.

      Now, historically, you couldn't break them down.

      You were allocated one and that was it.

      Classless into domain routing or SIDA, let's just take networks and break them down.

      It defines a way of expressing the size of a network.

      And this is called a prefix.

      An example of this is this /16 network.

      So 10.16.0.0/16.

      In this case, 16 is the prefix.

      Now, you might spot that this is actually inside of the class A network space.

      So class A is between 0.anything and 127.anything.

      And you might also spot that it's in the private class A address space.

      So 10.anything is a private set of addresses.

      But this is only actually a subset of this wider network.

      10.0.0.0/8 would be the full 10.anything range.

      So /8 is the same as a class A network.

      The first octet is the network and the rest is available for hosts or subnetting.

      This is a /16, which means that the first two octets are the network, so 10.16, and the rest is available for hosts or subnetting.

      So this /16 is a smaller network within the 10.0.0.0/8, bigger class A network.

      The larger the prefix value, the smaller the network.

      And that's a useful one to remember.

      Subnetting is a pretty complex process to do well, but you can learn the basics easily enough.

      Take this network as an example.

      10.16.0.0/16.

      If you watched my network fundamental series of videos, you'll know that 10.16.0.0/16 is a range which starts at 10.16.0.0 and ends at 10.16.255.255. /16 tells us that the network part is the first two octets, so 10 and 16.

      The network range, therefore, is 0.0 to 255.255 in the hosts part of the IP address.

      Now, let's say that we were allocated this range within our organization, but we needed to break it down into multiple networks.

      So rather than one large network, let's say, for example, we needed four smaller networks.

      But what we can do is to subnet this network.

      We can break it down.

      All we do is that we break this network into two.

      The single /16 becomes two /17 networks.

      The first network starts at the starting point of the original network and ends at the halfway point, the point at which the second network starts.

      So the first subnetwork is 10.16.0.0 through to 10.16.127.255, so the halfway point.

      And the second network starts at 10.16.128.0 and goes through to 10.16.255.255.

      So one /16 network is the same as two /17 networks.

      But now we can use these two networks for different things within our organization.

      Now, we could follow the same process again if we needed more subnets.

      For now, we could keep the first /17 network at the top in red, but break the bottom one in green into two networks, so two /18 networks.

      The method would be the same, so the first subnetwork, so bottom left, would start at the starting point of the original bottom network, so 10.16.128.0.

      The second smaller network, so bottom right, would start at the midway point of the original network.

      So these two networks are both /18, which are half the size of the original /17.

      And this gives us three subnets, a /17 at the top and two smaller /18s at the bottom.

      We could follow the same process again.

      Remember, our target here is for subnets, so we can break down the top /17 network into two /18 networks.

      The upper left /18 network starts at the starting point of the previous /17 network, and it ends at the halfway point.

      The upper right /18 network starts at the midpoint and goes through to the end.

      So this is how subnetting and side-up works.

      The entire internet is a /0 network.

      That's why 0.0.0.0, which you'll see as a default route, matches the entire internet.

      All the way through to a /8, which is a class A network, /16, which is a class B network, and /24, which is a class C network.

      And then all the way through to /32, which represents a single IP address.

      Now this process will become much clearer once you start using this in lab or production environments.

      Generally, when you perform subnetting, you'll be breaking down a larger network into two, four, eight, or more smaller networks, always breaking into two and then into two again.

      But while it is unusual, it is possible to have odd numbers.

      You can break a network into two and then break only one half of that into two more, and this gives you three subnets.

      And this is the example at the top right of the screen.

      Now this is unusual, but it doesn't break any rules.

      Subnetting is the process of taking a larger network and breaking it down into more smaller networks, each of which has a higher prefix, which means a smaller network.

      So now that you know the high-level process, I've gone through it graphically.

      Let's take a look at this in a little bit more detail before we finish.

      We don't use the same example.

      And plus before, only now with more detail.

      So we start with a /16 network, 10.16.0.0.

      Assuming we need four smaller networks, the starting point is to calculate the start and end of this network range.

      In this case, 10.16.0.0/16 starts at 10.16.0.0, and finishes at 10.16.255.255.

      So we know that any /17 networks will be half of this size.

      So step two is to split the original range into two.

      The first /17 network starts at the starting point of the original network, so 10.16.0.0, and ends halfway through at the original range, so 10.16.127.255.

      So 10.16.0.0/17 means 10.16.0.0 through to 10.16.127.255.

      The second smaller network starts at the midpoint, so 10.16.128.0/17, so this starts at 10.16.128.0, and ends at 10.16.255.255.

      You've split the original /16 into two.

      You've created two smaller /17 networks, each of which occupies half of the original address space.

      Now, further splits follow the same process.

      Each smaller network has a higher prefix value, and is half the size of the parent's network range.

      Its first smaller network starts at the same starting address and finishes halfway, and the second one starts at the halfway point and finishes at the end.

      In this case, we have 10.16.128.0/18, and 10.16.192.0/18.

      Both of these are within the larger /17 range of 10.16.128.0/17.

      If you just think about this process as splitting the network range in half, you're going to create two smaller networks, one which uses the first half and one which uses the second half.

      And we can do the same process with the upper subnet, so 10.16.0.0/17, so the network in red.

      We can split that range in half, creating two smaller networks.

      We've got 10.16.0.0/18, and 10.16.64.0/18.

      Now, becoming proficient with this process just takes time.

      You need to understand how to calculate IP addresses, how subnet masks and prefixes work, and then you can just follow this process step by step to break down large networks into more and more smaller subnets.

      Eventually, you won't even need to calculate it at all.

      It will just become instinctive.

      I know at this point it might seem like a fair distance off, but I promise it will happen.

      Now, at this point, that's everything I wanted to cover in this lesson.

      I know it's been a lot of theory.

      Go ahead and finish the video.

      I'm on your ready.

      I look forward to you joining me in the next video.

    1. IP Address Space & Subnetting - PART1

      Welcome back and welcome to another video of this Network Fundamental series where I'll be discussing IP addressing and IP subnetting.

      Now we've got a lot to cover, so let's jump in and get started.

      IP version 4 addressing has been around since the early days of the internet.

      In fact, it was standardized in 1981 via the RFC 791 document, which is attached to this lesson if you want to take a look.

      Now it's still the most popular network layer protocol in use on the internet.

      IP version 4 addresses occupy a range from 0.0.0.0 to 255.255.255.255.

      And this is just under 4.3 billion IP addresses.

      Now that sounds like a lot, but with a current world population around the 8 billion mark, that's less than one IP version 4 address per person.

      Now the address space was originally fully managed by an organization called IANA, which is the internet assigned numbers authority.

      More recently though, parts of the address space have been delegated to regional authorities such as RIPE, ARIN and APNIC.

      Now the key thing to understand is that with a few exceptions, IP version 4 addressing is allocated, and that means that you have to be allocated public IP version 4 addresses in order to use them.

      You can't simply pick a random address and expect it to work on the public internet without significant issues.

      Now there is part of the address space which is private, and that's the addresses which are generally used within home networks, business networks and cloud platforms such as AWS or Azure.

      The private address space can be used and reused freely.

      So now you know there are 4.294 billion IP version 4 addresses.

      You know they start at 0.0.0.0 and end at 255.255.255.255.

      Now historically, this range was divided into multiple smaller ranges which are allocated for specific functions.

      First, the class A address space which starts at 0.0.0.0 and ends at 127.255.255.255.

      Now this range contains 128 networks, each of which has 16.7 million addresses.

      So these networks are 0.anything which is reserved 1.anything, 2.anything, all the way up to 127.anything.

      The first octet denotes the network with the remaining octets available for hosts or for subnetting as we'll cover later in this video.

      So this class of IP addresses, so class A, these were generally used for huge networks and historically these were allocated to huge businesses or organisations which had an internet presence in the early days of the internet.

      So businesses like Apple, the Ford Motor Company, the US Postal Service or various parts of the US military.

      Many of those organisations have since given up those ranges and these are now allocated to the regional managers of the IP address space for allocation to users in that region.

      Now next we have the class B address space and this starts at 128.0.0.0 and it ends at 191.255.255.255.

      Now this part of the IP address space offers a total of 16,384 networks, each of them containing 65,536 IP addresses.

      So this space was typically used for larger businesses which didn't need a class A allocation.

      Like with addresses in the class A space, these are now generally allocated to the regional authorities and they manage them and allocate them out to any organisation who requests and can justify addresses in this range.

      Now these networks take the format of 128.0.anything, 128.1.anything, 128.2.anything and then all the way through to 191.253.anything, 191.254.anything and then finally 191.255.anything.

      Now with this range of IP addresses so class B, the first two octets are for the network and the last two are for the organisation to assign to devices or to subnet into smaller networks and we'll be talking about that later in this video.

      Next we have the class C range which starts at 192.0.0.0 and ends at 223.255.255.255.

      Now this range provides over 2 million networks, each containing 256 IP addresses.

      So examples of this range include 192.0.1.anything and 192.0.2.anything.

      With class C networks, the first three octets denote the network and the remaining is available for hosts or for subnetting.

      Class C networks are historically used for smaller businesses who required an IP version 4 presence but weren't large enough for class B or class A addressing and these two are generally now allocated and controlled by regional authorities.

      Now there are two more classes, class B and class E, but these are beyond the scope of what this video covers.

      Class B is used for multicast and class E is reserved, so I'll cover those at another time.

      Now within this public IP version 4 space, certain networks are reserved for private use and this means you can use them however you want, but they aren't roundable across the public IP version 4 internet so these can only be used for private networks or cloud platforms such as AWS who often use them for private networking.

      So let's take a look at those.

      Private IP addresses are defined within a standards document called RFC1918 and this document defines three ranges of IP version 4 addresses which you're free to use internally.

      Now these can't be routed across the internet, but you can use them as you choose internally and as often as required and this is one reason why network address translation is needed to translate these private addresses into publicly roundable addresses so they can communicate with the internet and I cover network address translation in a separate video.

      The first private range is a single class A network which starts at 10.0.0.0 and ends at 10.255.255.255 and it provides a total of 16.7 million IP version 4 addresses.

      Now this private range is often used within cloud environments and it's generally chopped up into smaller subnetworks which I'll be covering later on in this video.

      The next private range is from 172.16.0.0 through to 172.31.255.255.

      Now this is a collection of class B networks, 16 of them to be precise so you have 172.16.anything, 172.17.anything, 172.18.anything and so on all the way through to 172.31.anything.

      Now each of these networks contains 65,536 addresses and in AWS one of these private ranges 172.31 is used for the default VPC and again these networks are generally broken into smaller subnetworks when used.

      Lastly we have 192.168.0.0 to 192.168.255.255 and this range is 256 class C networks so that means 192.168.0.anything, 192.168.1.anything, 192.168.2.anything and so on all the way through to 192.168.255.anything so it provides 256 networks each containing 256 addresses and this range is generally used within home and small office networks so my home network for example uses one of these ranges for all of my devices and my router provides NAT or network address translation services in order to allow them to access the public internet.

      Now with any of these ranges you can use them however you want, you can reuse them you can break them up into smaller networks but in all cases you should try and avoid using the same one multiple times.

      If you ever need to connect private networks together and they use the same network addressing even if it's private you will have trouble configuring that communication.

      Where possible you should always aim to allocate non-overlapping ranges to all of your networks.

      So now let's move on to talk about IP version 6 and the differences between it and IP version 4.

      So to fully understand the need for IP version 6 and the differences it's useful to start with a representation of the IP version 4 address space.

      So we know now that this historically has been broken up into three common classes of IP addresses I've just talked about those.

      All of these IP addresses except for a few exceptions are publicly routable meaning if you have one of them configured on a device then you can communicate with another device which is also using a public IP version 4 address.

      Now I've also just talked about how part of this IP address space is dedicated for use for private networking but this in its entirety is the IP version 4 address space and in total there are 4,294,967,296 IP addresses so this is the total number of IP version 4 addresses available for use.

      Now this might sound like a lot but that's less than one person alive on the planet today and how many of us have a mobile phone and the computer so we have multiple devices already.

      What about providers like AWS who have huge public IP addressing requirements?

      Well IP version 6 was designed to solve this problem.

      The problem that we have far too few IP version 4 addresses and at this point we've essentially exhausted the supply.

      With IP version 6 we have more IP addresses to use and to fully appreciate this I want to change the perspective.

      This doesn't even do the scale justice but any smaller and you won't be able to see the blue square which now represents the total IP version 4 address space.

      Imagine the blue square is actually several thousand times smaller than it is now and with that in mind this is how the IP version 6 address space looks in comparison.

      The entire IP version 4 address space available on the public IP version 4 internet is just over 4 billion IP version 4 addresses.

      With IP version 6 the entire address space is 340 trillion trillion trillion addresses.

      Now humans are bad with large numbers but to put this into perspective it means that there are 670 quadrillion IP version 6 IP addresses per square millimeter of the Earth's surface or to put it another way 50 octillion IP addresses per human alive today or 79 octillion IP version 4 internet's worth of addressing within the IP version 6 address space.

      Now think about that for a moment it's enough to give you a headache there are 79 octillion sets of 4.3 billion IP addresses in the IP version 6 address space.

      That is an incredibly large number.

      Now I don't expect you to remember all of these numbers.

      What I want to do is make you comfortable with just how many IP version 6 addresses are available for use.

      With IP version 6 the concept of IP addresses as a valuable commodity just goes away.

      There are so many of them that you essentially don't require detailed IP management anymore it's just not a scarce resource.

      So that's IP version 6.

      Next I want to talk about subnetting from an IP version 4 perspective because this is a really useful skill that you should have when you start using a cloud environment.

      Okay so this is the end of part one of this lesson.

      It was getting a little bit on the long side and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part 2 will be continuing immediately from the end of part one.

      So go ahead complete the video and when you're ready join me in part two.

    1. Welcome back, this is part two of this lesson.

      We're going to continue immediately from the end of part one, so let's get started.

      The principle of Dynamic NAT is similar to static except that devices are not allocated a permanent public IP.

      Instead, they're allocated one temporarily from a pool.

      Let's say that we have two public IP addresses available for use, 52.95.36.66 and 67.

      But we have four devices on the left and all of them at some time need to use public addressing.

      So we can't use static NAT because we don't have enough public IP addresses.

      With Dynamic NAT, the public to private mapping is allocation-based, so it's allocated as required.

      Let's look at an example.

      Let's assume that the server on the top left is trying to access the CAT API.

      While it creates a packet, the source IP address is itself and the destination IP is the CAT API, which is 1.3.3.7.

      So it sends this packet and again the router in the middle is the default gateway for anything which is not local.

      As the packet passes through the router or the NAT device, it checks if the private IP has a current allocation of public addressing from the pool and if it doesn't and one is available, it allocates one dynamically and on a temporary basis.

      In this case, 52.95.36.67 is allocated on a temporary basis.

      So the packets source IP address is translated to this address and the packets are sent onto their final destination.

      The CAT API is able to send the response traffic back using this public IP allocation.

      So this process is the same so far as if we were using static NAT.

      But because Dynamic NAT allocates addressing on a dynamic and temporary basis, multiple private devices can share a single public IP as long as there is no overlap, so as long as the devices use the allocations at different times.

      In this case, the upper laptop is accessing the CATFLIX public service using 52.95.36.66 and then afterwards, the lower laptop is using the same public IP address to access the dogflix application.

      With Dynamic NAT, because the shared public pool of IP addresses is used, it is possible to run out of public IP addresses to allocate.

      If the bottom server attempts to access the public internet, when there are no IPs available in the pool to allocate, then this access will fail.

      Now this type of NAT is used when you have less public IPs than private ones, but when all of those private devices at some time need public access, which is bi-directional.

      Now the last type of NAT which I want to talk about is the one which you're probably familiar with.

      This is port address translation.

      This is the type of NAT you likely use on your home network.

      Port address translation is what allows a large number of private devices to share one public address.

      It's how the AWS NAT gateway functions within the AWS environment.

      It has a many to one mapping architecture.

      So many private IP version 4 addresses are mapped onto one single public IP version 4 address.

      Let's step through an example because this will make it easier to understand.

      The example we'll be using is three private devices on the left, all wanting to access Catflix on the right, which has a public IP of 1.3.3.7, and is accessed using TCP port 443, which in this case is HTTPS.

      And to make things easier, I'll be colour coding the laptops, so red for the top, purple for the middle, and yellow at the bottom.

      Now the way the port address translation or PAT works is to use both IP addresses and ports to allow for multiple devices to share the same public IP.

      Every TCP connection, in addition to a source and destination IP address, has a source and destination port.

      The destination port for outgoing connections is important because that's what the service runs on.

      In this case, Catflix uses the destination port of 443.

      The source port, this is randomly assigned by the client.

      So as long as the source port is always unique, then many private clients can use the same public IP.

      Let's assume that the public IP address at this NAT device is 52.95.36.67.

      So at this point, let's say that the top laptop, so the red laptop, generates a packet, and the packet is going to Catflix.

      So its destination IP address is 1.3.3.7, and its destination port is 443.

      Now the source IP of this packet is itself, so the laptop's private IP address, and the source port is 32.768, which is a randomly assigned ephemeral port.

      So this packet is routed through the NAT device on its way to the internet, and in transit, the NAT device records the source IP and the original source private port, and it allocates a new public source port, which in this case is 1.3.3.7.

      It records this information inside a NAT table, and it adjusts the packet or translates the packet, so that its source IP address is this single public address which the NAT device uses, and the source port is this newly allocated source port, which is now recorded within the NAT device.

      And this newly adjusted packet is forwarded on to Catflix.

      If the middle purple laptop did the same thing, then the same process would be followed.

      It would record all of this information, it would allocate a new public source port, and it would translate the packet, so adjust the packet's source IP address and the source port, to these newly defined values.

      Now if the bottom laptops or the yellow laptop generated a packet, note how this time the source port, which is randomly assigned, is the same source port that the top or red laptop is using for the same connection.

      But the same process would be followed.

      The NAT device would pick a unique source port to allocate, and it would translate this packet.

      It would change the source IP address from the private IP to the single public IP, and it would change the source port of 32.768 to a unique new source port, in this case, 1.3.3.9.

      Now normally the reason that only one device can use the same public IP is because these source ports are randomly assigned.

      If multiple devices communicate with the same destination service, using the same destination port, and they happen to use the same source port, then it will look like the same connection.

      What the NAT device is doing is creating this, a NAT table.

      The table is updated with the original private IP and private source port, and the new source IP, which is the public IP address of the NAT device, and then the newly allocated public source port.

      This means that when response data comes back, this table can be referenced to ensure that the packet reaches its destination.

      So when return traffic occurs, it will be from TCP port 443, with a source IP address of 1.3.3.7.

      The destination IP will be the NAT device's public IP, so 52.95.36.67.

      And the destination port will be the public source port that NAT device initially translated to.

      Let's say in this case, the public source port is 1.3.3.7, which represents the session of the top left laptop.

      So for return traffic, if an entry is present in the NAT table, the NAT device translates the public IP and public port, which are the destination IP and port to the original IP, which is 10.0.0.42 to the top laptop, and 32.768, which is the original source port number.

      Now it's worth pausing and making sure that you really understand how this process works, because it's how your home route works, and it's how the NAT gateway within AWS works.

      Once you understand it, you'll understand why, with port address translation, you can't initiate traffic to these private devices, because without an entry in the NAT table, the NAT device won't know to which device traffic should be translated and forwarded to.

      Now, I hope all of this has made sense, and you understand all of the different types of NAT.

      NAT is a great topic to understand, because it's one of those things which is used constantly within most architectures, cloud platforms, and even home and business networks.

      Now, that's everything I wanted to cover, though, so go ahead and complete this video, and when you're ready, I'll look forward to you joining me in the next video.

    1. Welcome back.

      In this lesson, I want to talk about network address translation known as NAT.

      NAT is used within home networks, business networks and cloud environments such as AWS.

      If you have a thorough understanding of NAT and how it works, it will make using any type of networking including AWS so much easier.

      Now I want to keep this video as brief and efficient as possible to let you jump in and take a look at exactly what NAT is and how it works.

      So NAT is a process which is designed to address the growing shortage of IP version 4 addresses.

      IP version 4 addresses are either publicly routable or they fall within the private address space of IP version 4.

      Publicly routable addresses are assigned by a central agency and regional agencies which in turn assign them to ISPs and these ISPs allocate them to business or consumer end users.

      An IP version 4 publicly routable addresses have to be unique in order to function correctly.

      Private addresses such as those in the 10.0.0.0 range can be used in multiple places but can't be routed over the internet.

      And so to give internet access to private devices, we need to use network address translation.

      In addition to this, NAT also provides some additional security benefits which I'll be covering soon.

      Now there are actually multiple types of NAT which I'm going to cover and all of them, they translate private IP addresses into public IP addresses so the packets can flow over public internet and then translate back in reverse.

      So that internet-based hosts can communicate back with these private services.

      So that's the high level function of NAT, which each type of NAT handles this process differently.

      First we've got static NAT which is where you have a network of private IP version 4 addresses and can allocate a public IP version 4 address to individual private IP addresses.

      So the static NAT device translates from one specific private address to one specific public address in effect giving that private address access to the public internet in both directions.

      And this is how the internet gateway within AWS works which I'll be covering in another video.

      Static NAT is what you would use when certain specific private IP addresses need access to the internet using a public ID and where these IPs need to be consistent.

      Dynamic NAT is similar but there isn't this static allocation.

      Instead you have a pool of public IP addresses to use and these are allocated as needed so when private IP addresses attempt to use the internet for something.

      This method of NAT is generally used when you have a large number of private IP addresses and want them all to have internet access via public IPs but when you have less public IP addresses than private IP addresses and you want to be efficient with how they're used.

      Then lastly we have port address translation and this is where many private addresses are translated onto a single public address.

      This is likely what your home internet route does, you might have many devices so laptops, computers, tablets, phones and all of those will use port address translation also known as overloading to use a single public IP address.

      Now this method as the name suggests uses ports to help identify individual devices and I'll cover in detail how this method works later in this video.

      This is actually the method that the NAT gateway or NAT instances use within AWS if you have any AWS experience then you'll recognise this process when I'm talking about the NAT gateway and NAT instances in a separate video.

      Now NAT is a process that only makes sense for IP version 4.

      Since IP version 6 adds so many more addresses we don't need any form of private addressing and as such we don't need translation.

      So try and remember this one IP version 6 generally means you don't need any form of network address translation.

      Okay so now I want to step through each of the different methods graphically so you can understand how they work and I'm going to be starting with static network address translation or static NAT.

      To illustrate this we want to use a visual example so let's start with a router and NAT gateway in the middle and a private network on the left and then a public network on the right.

      We have a situation where we have two devices in the private network, a server and a laptop and both of these need access to external services and let's use the example of Netflix and the CAT API.

      So the devices on the left they are private and this means they have addresses in the IP version 4, private address space in this case 10.0.0.10 for the server toward the top and 10.0.0.42 for the laptop toward the bottom.

      This means that these two devices packets that they generate cannot be routed over the public internet because they only have private addressing.

      Now the CAT API and Netflix both have public IP addresses in the case of the CAT API this is 1.3.7.

      So the problem we have with this architecture is that the private addresses can't be routed over the public internet because they're private only.

      The public addresses of the public internet-based services can't directly communicate with these private addresses because public and private addresses can't communicate over the public internet.

      What we need is to translate the private addresses that these devices have on the left to public IP addresses which can communicate with the services on the right and vice versa.

      Now with static NAT the router or NAT device maintains what's known as a NAT table and in the case of static network address translation the NAT table stores a one-to-one device mapping between private IP and public IP.

      So any private device which is enabled will have a dedicated, allocated public IP version 4 address.

      Now the private device won't have the public IP address configured on it, it's just an allocation.

      So let's say that the laptop on the bottom left wants to communicate with Netflix.

      Well to do so it generates a packet as normal.

      The source IP of the packet is the laptop's private IP address and the destination IP of the packet is one of Netflix's IPs.

      Let's say for this example we get issues in DNS.

      Now the router in the middle is the default gateway for any destinations so any IP packets which are destined for anything but the local network are sent to this router.

      Let's assume that we've allocated a public IP address to this laptop of 52.95.36.67.

      So there's an entry in the NAT table containing 10.0.0.42 which is the private address and 52.95.36.67 which is the public address and these are statically mapped to one another.

      In this case as the packet passes through the NAT device the source address of the packet is translated from the private address to the applicable public address and this results in this new packet.

      So this new packet still has Netflix as the destination but now it has a valid public IP address as the source.

      So because we've allocated this bottom laptop a public IP address as the packet moves through the NAT device the NAT device translates the source IP address of this packet from the private laptop's IP address to the allocated public address.

      So this is an example of static NAT and for anyone who's interested in AWS this is the process which is performed by the internet gateway so one to one static network address translation.

      Now this process works in a similar way in both directions.

      So let's say that the API client so the server on the top left wants to communicate with the CAT API.

      Well the same process is followed it generates a packet with the destination IP address of the CAT API and it sends it as it's passing through the NAT device the router replaces or translates the source address from the private IP address to the allocated public address.

      In this case 52.95.36.68.

      The CAT API once it receives the packet sees the source as this public IP so when it responds with data its packet has its IP address as the source and the previous public IP address as the destination the one which is allocated to the server on the top left.

      So it sends this packet back to this public IP and remember this public IP is allocated by the NAT device in the middle to the private device at the top left of the API client.

      So when this packet arrives at the NAT device the NAT table is checked it sees the allocation is for the server on the top left and so this time for incoming traffic the destination IP address is updated to the corresponding private IP address and then the packet is forwarded through to the private server.

      This is how static networks public IPs are allocated to private IPs.

      For outgoing traffic the source IP address is translated from the private address to the corresponding public address and for incoming traffic the destination IP address is translated from the allocated public address through to the corresponding private IP address.

      Now at no point are the private devices configured with a public IP.

      They always have private IP addresses and just to reiterate this is how the AWS internet gateway works which you'll either already know about or will learn about in a different video.

      So this is static NAT now let's move on to dynamic NAT.

      Okay so this is the end of part one of this lesson.

      It was getting a little bit on the long side and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part two will be continuing immediately from the end of part one.

      So go ahead complete the video and when you're ready join me in part two.

    1. Welcome back, this is part two of this lesson. We're going to continue immediately from the end of part one. So let's get started.

      Now that you know the structure of a segment, let's take a look at how it's used within TCP.

      Let's take a few minutes to look at the architecture of TCP.

      TCP, like IP, is used to allow communications between two devices.

      Let's assume a laptop and a game server.

      TCP is connection-based, so it provides a connection architecture between two devices.

      And let's refer to these as the client and the server.

      Once established, the connection provides what's seen as a reliable communication channel between the client and the server, which is used to exchange data.

      Now let's step through how this actually works, now that you understand TCP segments.

      The actual communication between client and server, this will still use packets at layer three.

      We know now that these are isolated.

      They don't really provide error checking, any ordering, and they're isolated, so there's no association between each other.

      There's no connection as such.

      Because they can be received out of order, and because there are no ports, you can't use them in a situation where there will be multiple applications or multiple clients, because the server has no way of separating what relates to what.

      But now we have layer four, so we can create segments.

      Layer four takes data provided to it and chops that data up into segments, and these segments are encapsulated into IP packets.

      These segments contain a sequence number, which means that the order of segments can be maintained.

      If packets arrive out of order, that's okay, because the segments can be reordered.

      If a packet is damaged or lost in transit, that's okay, because even though that segment will be lost, it can be retransmitted, and segments will just carry on.

      TCP gives you this guaranteed reliable ordered set of segments, and this means that layer four can build on this platform of reliable ordered segments between two devices.

      It means that you can create a connection between a client and the server.

      In this example, let's assume segments are being exchanged between the client and the game server.

      The game communicates to a TCP port 443 on the server.

      Now, this might look like this architecturally, so we have a connection from a random port on the client to a well-known port, so 443 on the game server.

      So between these two ports, segments are exchanged.

      When the client communicates to the server, the source port is 23060, and the destination port is 443.

      This architecturally is now a communication channel.

      TCP connections are bi-directional, and this means that the server will send data back to the client, and to do this, it just flips the ports which are in use.

      So then the source port becomes TCP443 on the server, and the destination port on the client is 23060.

      And again, conceptually, you can view this as a channel.

      Now, these two channels you can think of as a single connection between the client and the server.

      Now, these channels technically aren't real, they're created using segments, so they build upon the concept of this reliable ordered delivery that segments provide, and give you this concept at a stream or a channel between these two devices over which data can be exchanged, but understand that this is really just a collection of segments.

      Now, when you communicate with the game server in this example, you use a destination port of 443, and this is known as a well-known port.

      It's the port that the server is running on.

      Now, as part of creating the connection, you also create a port on your local machine, which is temporary, this is known as the ephemeral port.

      This tends to use a higher port range, and it's temporary.

      It's used as a source port for any segments that you send from the client to the server.

      When the server responds, it uses the well-known port number as the source, and the ephemeral port as the destination.

      It reverses the source and destination for any responses.

      Now, this is important to understand, because from a layer 4 perspective, you'll have two sets of segments, one with a source port of 23060 and a destination of 443, and ones which are the reverse, so a source port of 443, and a destination of 23060.

      From a layer 4 perspective, these are different, and it's why you need two sets of rules on a network ACL within AWS.

      One set for the initiating part, so the laptop to the server, and another set for the response part, the server to the laptop.

      When you hear the term ephemeral ports or high ports, this means the port range that the client picks as the source port.

      Often, you'll need to add firewall rules, allowing all of this range back to the client.

      Now, earlier, when I was stepping through TCP segment structure, I mentioned the flags field.

      Now, this field contains, as the name suggests, some actual flags, and these are things which can be set to influence the connection.

      So, Finn will finish a connection, Akk is an acknowledgement, and Sin is used at the start of connections to synchronize sequence numbers.

      With TCP, everything is based on connections.

      You can't send data without first creating a connection.

      Both sides need to agree on some starting parameters, and this is best illustrated visually.

      So, that's what we're going to do.

      So, the start of this process is that we have a client and a server.

      And as I mentioned a moment ago, before any data can be transferred using TCP, a connection needs to be established, and this uses a three-way handshake.

      So, step one is that a client needs to send a segment to the server.

      So, this segment contains a random sequence number from the client to the server.

      So, this is unique in this direction of travel for segments.

      And this sequence number is initially set to a random value known as the ISN or initial sequence number.

      So, you can think of this as the client saying to the server, "Hey, let's talk," and setting this initial sequence number.

      So, the server receives the segment, and it needs to respond.

      So, what it does is it also picks its own random sequence number.

      We're going to refer to this as SS, and it picks this as with the client side randomly.

      Now, what it wants to do is acknowledge that it's received all of the communications from the client.

      So, it takes the client sequence number, received in the previous segment, and it adds one.

      And it sets the acknowledgement part of the segment that it's going to send to the CS plus one value.

      What this is essentially doing is informing the client that it's received all of the previous transmission, so CS, and it wants it to send the next part of the data, so CS plus one.

      So, it's sending this segment back to the client.

      It's picking its own server sequence, so SS, and it's incrementing the client sequence by one, and it sends this back to the client.

      So, in essence, this is responding with, "Sure, let's talk."

      So, this type of segment is known as a SIN-AC.

      It's used to synchronize sequence numbers, but also to acknowledge the receipt of the client sequence number.

      So, when the first segment was called a SIN, to synchronize sequence numbers, the next segment is called a SIN-AC.

      It serves two purposes.

      It's also used to synchronize sequence numbers, but also to acknowledge the segment from the client.

      The client receives the segment from the server.

      It knows the server sequence, and so, to acknowledge to the server that it's received all of that information, it takes the server sequence, so SS, and it adds one to it, and it puts this value as the acknowledgement.

      Then it also increments its own client sequence value by one, and puts that as the sequence, and then sends an acknowledgement segment, containing all this information through to the server.

      Essentially, it's saying, "Autumn, let's go."

      At this point, both the client and server agree on the sequence values.

      The client has acknowledged the initial sequence value decided by the server, and the server has acknowledged the initial value decided by the client.

      So, both of them are synchronized, and at this point, data can flow over this connection between the client and the server.

      Now, from this point on, any time either side sends data, they increment the sequence, and the other side acknowledges the sequence value plus one, and this allows for retransmission when data is lost.

      So, this is a process that you need to be comfortable with, so just make sure that you understand every step of this process.

      Okay, so let's move on, and another concept which I want to cover is sessions, and the state of sessions.

      Now, you've seen this architecture before, a client communicating with the game server.

      The game server is running on a well-known port, so TCP 443, and the client is using an ephemeral port 23060 to connect with port 443 on the game server.

      So, response traffic will come up from the game server, its source port will be 443, and it will be connecting to the client on destination port 23060.

      Now, imagine that you want to add security to the laptop, let's say using a firewall.

      The question is, what rules would you add?

      What types of traffic would you allow from where and to where in order that this connection will function without any issues?

      Now, I'm going to be covering firewalls in more detail in a separate video.

      For now though, let's keep this high level.

      Now, there are two types of capability levels that you'll encounter from a security perspective.

      One of them is called a stateless firewall.

      With a stateless firewall, it doesn't understand the state of a connection.

      So, when you're looking at a layer 4 connection, you've got the initiating traffic, and you've got the response traffic.

      So, the initiating traffic in light with the bottom, and the response traffic in red at the top.

      With a stateless firewall, you need two rules.

      A rule allowing the outbound segments, and another rule which allows the response segments coming in the reverse direction.

      So, this means that the outbound connection from the laptop's IP, using port 23060, connecting to the server IP, using port 443.

      So, that's the outgoing part.

      And then the inbound response coming from the service IP on port 443, going to the laptop's IP on a femoral port 23060.

      So, the stateless firewall, this is two rules, one outbound rule and one inbound rule.

      So, this is a situation where we're securing an outbound connection.

      So, where the laptop is connecting to the server.

      If we were looking to secure, say, a web server, where connections would be made into our server, then the initial traffic would be inbound, and the response would be outbound.

      There's always initiating traffic, and then the response traffic.

      And you have to understand the directionality to understand what rules you need with a stateless firewall.

      So, that's a stateless firewall.

      And if you have any AWS experience, that's what a network access control list is.

      It's a stateless firewall which needs two rules for each TCP connection, one in both directions.

      Now, a stateless firewall is different.

      This understands the state of the TCP segment.

      So, with this, it sees the initial traffic and the response traffic as one thing.

      So, if you allow the initiating connection, then you automatically allow the response.

      So, in this case, if we allowed the initial outbound connection from the client laptop to the server, then the response traffic, the inbound traffic, would be automatically allowed.

      In AWS, this is how a security group works.

      The difference is that a stateless firewall understands level and the state of the traffic.

      It's an extension of what a stateless firewall can achieve.

      Now, this is one of those topics where there is some debate about whether this is layer four or layer five.

      Layer four uses TCP segments and concerns itself with ID addresses and port numbers.

      Strictly speaking, the concept of a session or an ongoing communication between two devices, that is layer five.

      It doesn't matter if this level item can by layer four and layer five anyway, because it's just easier to explain.

      But you need to remember the term stateless and the term stateful and how they change how you create security rules.

      For this point, that's everything I wanted to cover.

      So, go ahead and complete this video. And when you're ready, I'll look forward to you joining me in the next video of this series.

    1. Welcome back. In this part of the series, I'm going to be primarily covering the transport layer, which is layer 4 of the OSI model.

      I'm also going to be touching upon layer 5, which is the session layer of the OSI model, because there is some overlap for certain features, and so it's easier to cover them in one lesson.

      The transport layer runs over the top of the network layer and provides most of the functionality, which supports most of the networking, which we use day-to-day on the internet.

      The session layer runs on top of the transport layer, and many features, which you might use, are often mixed between these two layers.

      Now, as I've already mentioned, it's not generally worth the argument of deciding whether things are covered in layer 4 or layer 5, so I'd only explain both of these layers as one grouping of functionality.

      The OSI model is conceptual, after all, and many things exist between or across two different layers.

      Now, we've got a lot to cover, so let's jump in and get started.

      Before we get started with layer 4, I want to summarize the situation and limitations with layer 3.

      Now, we have a functional layer 3, which means that we can communicate between two devices, say a source and destination laptop, using a source and destination IP address.

      If both of these use public IP addresses, it doesn't matter where on the internet these devices are, layer 3 and IP routing will ensure that any packets generated and sent from the source laptop will move across any layer 2 networks between the source and destination.

      Let's say that using layer 3, the source laptop on the top generates 6 IP packets, and these are all destined for the destination laptop at the bottom right.

      The important thing to understand about layer 3 in this context is that each packet is a separate and isolated thing, and it's routed independently over the internet.

      It might be logical to assume that the packets arrive in the same state, so the same timing, the same order, and the same quality, but sadly, that's not true.

      In ideal conditions, yes, but generally, if you're communicating using only IP, then you're going to have intermittent network conditions, and that can result in a few cases where the arrival condition of packets is different than the condition when they were generated and sent.

      One of the first things which we might encounter is out-of-order arrival.

      In this case, where packet 3 arrives before packet 2, layer 3, specifically IP, provides no method to ensure the ordering of packet arrival.

      For applications which only used IP, this would mean complex logic would need to be built into the application to ensure packets could be sequenced in the same way, and this is not a trivial task.

      Because each packet is routed as an independent thing, it's possible packet 2 could have taken a slow, less efficient route, which is why it arrives later.

      This is a negative of layer 3, which can be fixed at layer 4.

      Another issue with layer 3 is that packets can just go missing.

      This can be due to network outages or network conditions, which cause temporary routing loops.

      Remember, when I talked about packet structure, I talked about the TTL field, which limited the number of hops a packet could go through.

      Well, if the number of hops exceeds this, then it will be discarded.

      With IP, there's no reliable method of ensuring packet delivery, and so it's a relatively regular occurrence that packets go missing.

      Now, network conditions can also cause delay in delivery, and for any latency-sensitive applications, this can cause significant issues.

      The key thing to keep in mind about layer 3, every packet is different.

      It's single, it's isolated.

      It's a different unit of data which is being routed across a layer 3 network using layer 2 networks as transit.

      What happens to one packet might not happen or might happen in a different way to another packet.

      Another limitation with layer 3, and this one is probably the one which has the most obvious effect, is that if you think back to the structure of IP packets, they have a source and destination field.

      They don't have anything beyond that to distinguish channels of communication.

      Packets from a source IP to a destination IP, they're all the same.

      You couldn't have two applications running on the source IP, communicating with two applications running on the destination IP, because there's no method of distinguishing between the applications.

      Any packet sent by one application would look to be the same as one sent by another.

      Think about what you're doing on your device right now.

      You might be watching a video.

      Do you have a web browser open doing something else?

      Do you have an SSH connection or email or any other application which uses the internet?

      This means multiple applications, and IP on its own offers no way to separate the packets for individual applications.

      This is something which is remedied at layer 4.

      Lastly, IP has no flow control.

      If a source device is transmitting packets faster than a destination device can receive them, then it can saturate the destination connection and cause loss of data, packets which will be dropped.

      Now with only layer 3, we wouldn't have anywhere near the flexibility required to have the internet function in the way that it does.

      For that, we need layer 4, and that's what I want to cover in this part of the lesson series.

      So what is layer 4 and how does it function?

      Let's take a look.

      So far, this is what we have network model-wise.

      We've discussed the physical layer which is layer 1 at the OSI model.

      This relates to how raw bit screen data is transmitted to or received from physical shared media.

      We've talked about layer 2 which adds identifiable devices, switches and media access control, but layer 2 ends with isolated layer 2 networks.

      In the previous part of this lesson series, I introduced layer 3 which adds IP addressing and routing, so packets can be routed from source to destination across multiple interconnected networks.

      Layer 4 builds on top of this.

      It adds two new protocols, TCP which stands for transmission control protocol and UDP which stands for user datagram protocol.

      Now both of these run on top of IP, and both of them add a collection of features depending on which one of them is used.

      Now if you've heard the term TCP/IP, that means TCP running on top of IP.

      At a high level, you would pick TCP when you want reliability, error correction and ordering of data.

      It's used for most of the important application layer protocols such as HTTP, HTTPS, SSH and so on.

      Now TCP is a connection-oriented protocol which means that you need to set up a connection between two devices and once set up, it creates a bidirectional channel of communications.

      UDP on your hand is faster because it doesn't have the TCP overhead required for the reliable delivery of data.

      This means that it's less reliable.

      Now there's a great joke about UDP.

      I will tell you about it, but you might not get it.

      Anyway, it's a good job my lessons are better than my jokes.

      In this lesson, I'm going to spend most of my time talking about TCP because it's used by more of the important protocols that you'll use day-to-day on the internet.

      But just know that both TCP and UDP, they both run on top of IP and they're used in the same way.

      They use IP as transit.

      TCP just offers a more reliable connection-oriented architecture whereas UDP is all about performance.

      So there's a simple trade-off.

      Now for this lesson series, as I talk about, I'm going to be focusing on TCP because that's what's used for most of the important upper layer protocol.

      So let's take a look at exactly how TCP works.

      TCP introduces something called segments.

      Now a segment is just another container for data like packets and frames before them.

      Segments are specific to TCP.

      Before we get started talking about the segments themselves, it's important to understand that segments are actually contained in which is known as encapsulated within IP packets.

      So let's say that we have a stream of packets.

      You know by now that these are all isolated packets.

      They're just pieces of data which are routed independently from source to destination.

      They're all treated separately.

      Well, TCP segments are placed inside packets and the packets carry the segments from their source to their destination.

      Segments don't have source or destination IP addresses because they use the IP packets for the transit from source to destination.

      This is all handled by layer 3.

      In this case, the internet protocol.

      TCP segments add additional capabilities to IP packets.

      Let's step through the structure of segments so that we can fully understand them.

      And I'm going to skip past a few attributes of segments just as I did with layer 3 because there are some parts which are less important or less situational.

      So I won't be covering either the options or padding fields within a segment.

      The first fields which I want to cover are the source and destination ports.

      In addition to the source and destination IP addresses that IP packets provide, TCP segments add source and destination ports.

      And this gives the combined TCP/IP protocol the ability to have multiple streams of conversations at the same time between two devices.

      When you open the AWS web interface, you're communicating from a port on your local machine to a port on the AWS servers, TCP port 443, which is HTTPS.

      Now because of port, you can have multiple streams of communication from your machine.

      One to AWS, one to Netflix, and one to this website where you're watching this video.

      At the other side, AWS can have multiple streams of communication to their servers.

      Each conversation is a unique combination of the source and destination IP, the source port, and the destination port.

      All of these four values together identify as a single conversation, a single communications channel.

      These two fields ought to allow the internet to function in a flexible way that it does.

      It's why SSH and HTTPS can exist on the same EC2 instance and why you can have multiple SSH connections open to the same EC2 instance if you wanted to.

      And I'll cover more on how this works as we move through this lesson.

      Now next, within the segment, is sequence.

      And the sequence number is incremented with each segment that's sent.

      And it's unique.

      It can be used for error correction if things need to be retransmitted.

      You can use to ensure that one IP pass is received and the TCP segments are pulled out.

      They can be correctly ordered.

      So the sequence number is already uniquely identifying the particular segment within a particular connection so that both sides can make observations about it.

      And the way that these observations are done is using app knowledgements.

      The app knowledgement field on the right, on the one side, can indicate that it's received up to and including a certain sequence number.

      Every segment which is transmitted needs to be acknowledged.

      Remember that TCP is a reliable protocol and so if the device is transmitting segment one, two, three, and four to another device, then the other device needs to acknowledge that it's received segment one, two, three, and four.

      And this is what the app knowledgement field is for.

      So sequence number and app knowledgement are used hand in hand.

      Next we have a field called flags and things.

      Now within a segment, there is an actual flags component which is nine bits.

      And this allows various controls over the TCP segments and the wider connection.

      Flags are used to close the connection of the synchronized sequence numbers, but there's also additional things like a data offset and some reserved space.

      So I thought this flags and things is essentially the flags plus a number of extra fields which I don't need to go into at this point in the lesson.

      Now next we've got the TCP window.

      And this is interesting.

      This defines the number of bytes that you indicate that you're willing to receive between app knowledgements.

      Once reached, the sender will pause until you acknowledge that amount of data.

      And this is how flow control is implemented.

      It lets the receiver control the rate at which the sender sends data.

      If you use a smaller window, it provides additional levels of control over how quickly you're sent data.

      Larger windows are more efficient because the header of a TCP segment takes up an amount of space and the smaller the window, the more headers are involved.

      So this window setting is quite important if you're using a TCP for practical reasons, but we don't need to go into too much detail in this lesson.

      Next we have checks on which is used for error checking.

      It means that a TCP layer is able to detect errors and can arrange for retransmission of the data as required.

      And then lastly, we have the urgent pointer.

      And this is a cool feature.

      Imagine if you have a data transfer application where 99% of data is the data being transferred and 1% is control traffic.

      So communication between the client and the server, coordinating the actual data transfer.

      While setting this field in a segment means that both sides can have separate processing.

      So the control traffic always takes priority within the communication.

      So any protocols which are latency sensitive and transfer data such as FTP and PellNet can use this field.

      Now all of these fields together are known as the TCP header.

      And the capacity of a TCP segment which remains is logically enough used for data.

      So that's a segment that are placed inside packets and transmitted by one network stack, specifically layer 4 of one network stack and received by another network stack using the layer 4 protocol.

      In this case TCP.

      Okay so this is the end of part 1 of this lesson.

      It was getting a little bit on the long side and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part 2 will be continuing immediately from the end of part 1.

      So go ahead, complete video and when you're ready join me in part 2.

    1. Welcome back, this is part three of this lesson. We're going to continue immediately from the end of part two. So let's get started.

      The address resolution protocol is used generally when you have a layer three packet and you want to encapsulate it inside a frame and then send that frame to a MAC address.

      You don't initially know the MAC address and you need a protocol which can find the MAC address for a given IP address.

      For example, if you communicate with AWS, AWS will be the destination of the IP packets.

      But you're going to be forwarding via your home router which is the default gateway.

      And so you're going to need the MAC address of that default gateway to send the frame to containing the packet.

      And this is where ARP comes in.

      ARP will give you the MAC address for a given IP address.

      So let's step through how it works.

      For this example, we're going to keep things simple.

      We've got a local network with two laptops, one on the left and one on the right.

      And this is a layer three network which means it has a functional layer two and layer one.

      What we want is the left laptop which is running a game and it wants to send the packets containing game data to the laptop on the right.

      This laptop has an IP address of 133.33.3.10.

      So the laptop on the left takes the game data and passes it to its layer three which creates a packet.

      This packet has its IP address as the source and the right laptop as the destination.

      So 133.33.3.10.

      But now we need a way of being able to generate a frame to put that packet in for transmission.

      We need the MAC address of the right laptop.

      This is what ARP or the address resolution protocol does for us.

      It's a process which runs between layer two and layer three.

      It's important to point out at this point that now you know how devices can determine if two IP addresses are on the same local network.

      In this case, the laptop on the left because it has its subnet mask and IP address as well as the IP address of the laptop on the right.

      It knows that they're both on the same network.

      And so this is a direct local connection.

      Routers aren't required.

      We don't need to use any routers for this type of communication.

      Now ARP broadcasts on layer two.

      It sends an ARP frame to all Fs as a MAC address.

      And it's asking who has the IP address 133.33.3.10 which is the IP address of the laptop on the right.

      Now the right laptop because it has a full layer one, two and three networks stack is also running the address resolution protocol.

      The ARP software sees this broadcast and it responds by saying I'm that IP address.

      I'm 133.33.3.10.

      Here's my MAC address ending 5B colon 7, 8.

      So now the left laptop has the MAC address of the right one.

      Now it can use this destination MAC address to build a frame, encapsulate the packet in this frame.

      And then once the frame is ready, it can be given to layer one and sent across the physical network to layer one of the right laptop.

      Layer one of the right laptop receives this physical orbit stream and hands it off to the layer two software also on the right laptop.

      Now it's layer two software reviews the destination MAC address and sees that it's destined for itself.

      So it strips off the frame and it sends the packet to its layer three software.

      Layer three reviews the packet, sees that it is the intended destination and it de-encapsulates the data.

      So strips away the packet and hands the data back to the game.

      Now it's critical to understand as you move through this lesson series, even if two devices are communicating using layer three, they're going to be using layer two for local communications.

      If the machines are on the same local network, then it will be one layer two frame per packet.

      But if you'll see in a moment if the two devices are remote, then you can have many different layer two frames which are used along the way.

      And ARP, or the address resolution protocol, is going to be essential to ensure that you can obtain the MAC address for a given IP address.

      This is what facilitates the interaction between layer three and layer two.

      So now that you know about packets, now that you know about subnet masks, you know about routes and route tables, and you know about the address resolution protocol or ARP, let's bring this all together now and look at a routing example.

      So we're going to go into a little bit more detail now.

      In this example, we have three different networks.

      We've got the orange network on the left, we've got the green network in the middle, and then finally the pink network on the right.

      Now between these networks are some routers.

      Between the orange and green networks is router one, known as R1, and between the green and pink networks is router two, known as R2.

      Each of these routers has a network interface in both of the networks that it touches.

      Routers are layer three devices, which means that they understand layer one, layer two, and layer three.

      So the network interfaces in each of these networks work at layer one, two, and three.

      In addition to this, we have three laptops.

      We've got two in the orange network, so device one at the bottom and device two at the top, and then device three in the pink network on the right.

      Okay, so what I'm going to do now is to step through two different routing scenarios, and all of this is bringing together all of the individual concepts which I've covered at various different parts of this part of the lesson series.

      First, let's have a look at what happens when device one wants to communicate with device two using its IP address.

      First, device one is able to use its own IP address and subnet mask together with device two's IP address, and calculate that they're on the same local network.

      So in this case, router R1 is not required.

      So a packet gets created called P1 with a D2 IP address as the destination.

      The address resolution protocol is used to get D2's MAC address, and then that packet is encapsulated in a frame with that MAC address as the destination.

      Then that frame is sent to the MAC address of D2.

      Once the frame arrives at D2, it checks the frame, hits the destination, so it accepts it and then strips the frame away.

      It passes the packet to layer three.

      It sees that it's the destination IP address, so it strips the packet away and then passes the game data to the game.

      Now all of this should make sense.

      This is a simple local network communication.

      Now let's step through a remote example.

      Device two communicating with device three.

      These are on two different networks.

      Device two is on the orange network, and device three is on the pink network.

      So first, the D2 laptop, it compares its own IP address to the D3 laptop IP address, and it uses its subnet mask to determine that they're on different networks.

      Then it creates a packet P2, which has the D3 laptop as its destination IP address.

      It wraps this up in a frame called F2, but because D3 is remote, it knows it needs to use the default gateway as a router.

      So for the destination MAC address of F2, it uses the address resolution protocol to get the MAC address of the local router R1.

      So the packet P2 is addressed to the laptop D3 in the pink network, so the packet's destination IP address is D3.

      The frame F2 is now addressed to the router R1 at MAC address, so this frame is sent to router R1.

      R1 is going to see that the MAC address is addressed to itself, and so it will strip away the frame F2, leaving just the packet P2.

      Now a normal network device such as your laptop or phone, if it received a packet which wasn't destined for it, it would just drop that packet.

      A router though, it's different.

      The router's job is to route packets, so it's just fine to handle a packet which is addressed somewhere else.

      So it reviews the destination of the packet P2, it sees that it's destined for laptop D3, and it has a route for the pink network in its route table.

      It knows that for anything destined for the pink network, then router R2 should be the next hop.

      So it takes packet P2 and it encapsulates it in a new frame F3.

      Now the destination MAC address of this frame is the MAC address of router R2, and it gets this by using the address resolution protocol or ARP.

      So it knows that the next hop is the IP address of router R2, and it uses ARP to get the MAC address of router R2, and then it sends this frame off to router R2 as the next hop.

      So now we're in a position where router R2 has this frame F3 containing the packet P2 destined for the machine inside the pink network.

      So now the router R2 has this frame with the packet inside.

      It sees that it's the destination of that frame.

      The MAC address on the frame is its MAC address, so it accepts the frame and it removes it from around packet P2.

      So now we've just got packet P2 again.

      So now router R2 reviews the packet and it sees that it's not the destination, but that doesn't matter because R2 is a router.

      It can see that the packet is addressed to something on the same local network, so it doesn't need to worry anymore about routing.

      Instead, it uses ARP to get the MAC address of the device with the intended destination IP address, so laptop D3.

      It then encapsulates the packet P2 in a new frame, F4, whose destination MAC address is that of laptop D3, and then it sends this frame through to laptop D3. laptop D3 receives the frame, D3 sees that it is the intended destination of the frame because the MAC address matches its MAC address.

      It strips off the frame, it also sees that it's the intended destination of the IP packet, it strips off the packet, and then the data inside the packet is available for the game that's running on this laptop.

      So it's a router's job to move packets between networks.

      Router's doing this by reviewing packets, checking route tables for the next hop or target addresses, and then adding frames to allow the packets to pass through intermediate layer 2 networks.

      A packet during its life might move through any number of layer 2 networks and be re-encapsulated many times during its trip, but normally the packet itself remains unchanged all the way from source to destination.

      A router is just a device which understands physical networking, it understands data link networking, and it understands IP networking.

      So that's layer 3, the network layer, and let's review what we've learned quickly before we move on to the next layer of the OSI model.

      Now this is just an opportunity to summarize what we've learned, so at the start of this video, at layer 2 we had media access control, and we had device to device or device to all device communications, but only within the same layer 2 network.

      So what does layer 3 add to this?

      Well it adds IP addresses, either version 4 or version 6, and this is cross network addressing.

      It also adds the address, resolution, protocol, or ARP, which can find the MAC address for this IP address or for a given IP address.

      Layer 3 adds routes, which define where to forward a packet to, and it adds route tables, which contain multiple routes.

      It adds the concept of a device called a router, which moves packets from source to destination, encapsulating these packets in different layer 2 frames along the way.

      This altogether allows for device to device communication over the internet, so you can access this video, which is stored on a server, which has several intermediate networks away from your location.

      So you can access this server, which has an IP address, and packets can move from the server through to your local device, crossing many different layer 2 networks.

      Now what IP doesn't provide?

      It provides no method of individual channels of communication.

      Layer 3 provides packets, and packets only have source IP and destination IP, so for a given two devices, you can only have one stream of communication, so you can't have different applications on those devices communicating at the same time.

      And this is a critical limitation, which is resolved by layers 4 and above.

      Another element of layer 3 is that in theory packets could be delivered out of order.

      Individual packets move across the internet through intermediate networks, and depending on network conditions, there's no guarantee that those packets will take the same route from source to destination, and because of different network conditions, it's possible they could arrive in a different order.

      And so if you've got an application which relies on the same ordering at the point of receipt as at the point of transmission, then we need to add additional things on top of layer 3, and that's something that layer 4 protocols can assist with.

      Now at this point we've covered everything that we need to for layer 3.

      There are a number of related subjects which I'm going to cover in dedicated videos, such as network address translation, and how the IP address space functions, as well as IP version 6, which in this component of the lesson series, we've covered how the architecture of layer 3 of the OSI model works.

      So at this point, go ahead and complete this video, and then when you're ready, I'll look forward to you joining me in the next part of this lesson series where we're going to look at layer 4.

    1. Welcome back, this is part two of this lesson.

      We're going to continue immediately from the end of part one, so let's get started.

      Now we talked about the source and destination IP address of these packets, so now let's focus on IP addressing itself.

      IP addressing is what identifies a device which uses layer 3 IP networking.

      Now I'll talk more about how IP addressing is decided upon and assigned in another video, for now I want you to fully understand the structure of an IP address.

      In this video I'll be focusing on IP version 4, because I have a separate video which will cover IP version 6 in depth.

      This is an IP address, 133.33.3.7.

      From a pure network connectivity point of view, if you have a valid IP version 4 address, you can send packets to 133.33.3.7 and they will at least start on the journey of getting to this destination.

      Now there might be blocks in the way, so firewalls or other security restrictions, all the IP could be offline, but packets will move from you over the internet on their way to this IP address.

      Now this format is known as dotted decimal notation.

      It's four decimal numbers from 0 to 255 separated by dots.

      So 133.33.3.7.

      Now all IP addresses are actually formed of two different parts.

      There's the network part which states which IP network this IP address belongs to, and then the host part which represents hosts on that network.

      So in this example the network is 133.33, and then the hosts on that network can use the remaining part of the IP.

      In this case 3.7 is one device on that network, a laptop.

      A really important part of understanding how your data gets from your location to a remote network is the given two IP addresses.

      How do you tell if they're on the same IP network or different IP networks?

      If the network part of the IP address matches between two different IP addresses, then they're on the same IP network.

      If not, they're on different IP networks.

      So you need to be able to calculate where you've an IP address, which part of that address is the network, and which part is the host.

      And by the end of this lesson you will know how to do that.

      Now IP addresses are not actually dotted decimal.

      That's how they're represented for humans.

      They're actually binary numbers.

      Each decimal part of the IP address is an 8-bit binary number.

      There are four of these per IP version 4 address, and this means that an entire IP address is 32 bits in size.

      So four sets of 8 bits, and each of these 8 bits is known as an octet.

      You might hear somebody refer to say the first and second octet of an IP address, and this is always read left to right.

      The first octet in this example is 1, 3, 3, or in binary 1, 0, 0, 0, 1, 0, 1.

      And the second octet is 33, which in binary is 0, 0, 1, 0, 0, 0, 0, 1.

      Now this binary conversion, this is not something which I'm going to cover in this lesson, but I will make sure there's a link attached to the lesson which shows you how to do it.

      It's just decimal to binary maths, and once you know how it's done, it's really easy to do, even in your head.

      Now I'm going to talk about how you can determine which IPs are on the same network next, but I wanted to introduce the format of IP addresses first.

      In this example, this IP address has what's known as a /16 prefix.

      This means that the first 16 bits represent the network, and the rest are for hosts.

      Now I don't really talk about how this works in detail coming up next.

      Because the first 16 bits are network, it means that the second IP address is 1, 3, 3, .33, .33, .37, because the network part of that matches is 1, 3, 3, .33, and it's on the same IP network.

      I'm going to detail coming up next how this calculation is done.

      For now, I want you to be comfortable knowing that if the network component of two IP addresses match, then devices are local.

      If they don't match, then devices are remote.

      That matters when we start covering IP routing.

      Now IP addresses are networks.

      These are either statically assigned by humans, and this is known as a static IP, or they're assigned automatically by machines.

      So service on your network running DHCP service software.

      Now DHCP stands for Dynamic Host Configuration Protocol, and this is something I'll be covering in detail in a separate video.

      On a network, IP addresses need to be unique, or bad things happen.

      Globally, in most cases, IP addresses need to be unique, or also bad things happen.

      So keep that in mind.

      Generally, when you're dealing with IP addresses, you want them to be unique, especially on your local network.

      Now let's talk about subnet masks, because these are what helps us determine if IP addresses are local to each other or remote.

      Subnet masks are a critical part of IP networking.

      They're configured on layer 3 interfaces, along with IP addresses.

      What's also configured on most network interfaces is a default gateway.

      This is an IP address on a local network, which packets are forwarded to, generally, if the intended destination is not a local IP address.

      Subnet masks are what allow an IP device to know if an IP address which it's communicating with is on the same network or not, and that influences if the device attempts to communicate directly on the local network, or if it needs to use the default gateway.

      On your home network, for example, your internet router is likely set as your default gateway, so when you browse to Netflix.com or interact with AWS because the IP addresses that you're talking to are not local, then packets from your machine are passed to your router, which is the default gateway.

      So let's say that we have an IP address, 133.33.3.7.

      Now this alone is just a single address.

      We don't know which part of it is the network and which part of it is the host component.

      I just finished talking about how IP addresses can match binary numbers.

      This IP address in binary is 1-0-0-0-1-0-1, so that's the first octet, and then 0-0-1-0-0-1, that's the second octet, and then 0-0-0-0-0-0-1-1, that's the third octet, and then finally 0-0-0-0-0-1-1-1, and that's the fourth octet, and that represents 133.33.3.7.

      So as a reminder, if we're dealing manually with subnet masks, and remember this is something that's generally performed in software by your networking stack, the first thing we need to do is convert the dotted decimal notation into a binary number.

      Now along with this IP address, we would generally also configure either statically or using DHCP, a subnet mask.

      In this example, the subnet mask that we have is 255.255.0.0 or /16, and these mean the same thing, and I'll show you why over the next few minutes.

      A subnet mask represents which part of the IP is for the network.

      It helps you, or more often a machine, know which part of an IP address is which.

      To use a subnet mask, you first have to convert it into binary, so 255.255.0.0 is this in binary.

      We convert it just like an IP address.

      So the first octet is all 1s, the second octet is all 1s, the third and fourth octet are all 0s.

      The /16, which is known as the prefix, this is just shorthand.

      It's the number of 1s in the subnet mask starting from the left.

      So /16 simply means 16 1s, which is the same as 255.255.0.0 when you convert that into binary.

      Now when you have the subnet mask in binary, anything with a 1 represents the network, anything with a 0 represents the host component.

      So if you overlay a subnet mask and an IP address, both of them in binary, it becomes really easy to tell which part is which.

      Something else which is really cool is that for a given network, you can calculate the start and end IP addresses of that network.

      Take for example, the IP address that's on screen now, so 133.33.3.7.

      Well we've converted that into binary and we've also converted the subnet mask of 255.255.0.0 also into binary.

      So that's in blue, right below the binary IP address.

      To calculate the start of the network, we begin with the network part of the IP address and then for the host part, we have all 0s.

      So let's look at what we've done.

      The subnet mask, where there are 1s, this is the network part.

      So we take the original IP address and where the subnet mask has 1s, that's the network part, so 133.33.

      Then for the part which is hosts, which is where the subnet mask shows 0s, then we have all 0s.

      This means that the network starting point is 133.33.0.0.

      Now to find the end, we take the network component of the IP address again, so where the subnet mask is all 1s, that's what we start with.

      And to work out the end of the network, we take the host component, so where the subnet mask is 0s, and we have all 1s in the IP address.

      So the ending part of this network is 133.33.255.255.

      So the starting address of a network is the network component of the IP address, identified with the subnet mask, and then all 0s for the host part of the IP address, and the ending address of the network is the network part of the IP address to start with, and then for the host component, we have all 1s.

      So this is how subnet masks work.

      They're used to identify which part of an IP address is the network part and which is the host part.

      As long as the network part for two different IP addresses is the same, then we know that both of those IP addresses are on the same IP network, and this is essential so that the machine can identify when it can send data directly on the same local network, or when IP routing needs to be used to transfer packets across different intermediate networks.

      So it's how your local device, your local laptop, knows to send packets to your internet router for Netflix or AWS, rather than trying to look for both of those systems locally on your local area network.

      And that's how a router makes that decision too, when it's looking where to forward packets to.

      So using subnet masks and IP addresses, it's how a lot of the intelligence of layer 3 is used.

      Now next, I want to spend some time looking at route tables and routes.

      Let's step through an example of data moving from you to AWS, and I want to keep focus for now on how a router makes a decision where to send data.

      Packets that you create for AWS will move from your house into your internet provider across the internet, potentially even between countries, and then finally arrive at AWS.

      Let's step through a simple example.

      So we start with our house on the left.

      Next, we have our internet provider known as an ISP or Internet Service Provider, and let's call this Meow ISP, and then we have three destination networks.

      We have AWS, our ISP's upstream provider, and then Netflix.

      Now we want to communicate with AWS, and so we create a packet on our local device, which has our IP address 1.3.3.7 as the source IP address, and it has a destination IP address of 52.217.13.37.

      Now you're going to have an internet router within your home, and this is where your device will send all of its data through.

      That router has what's known as a default route, which means all IP traffic is sent to it on its way to Meow ISP.

      Now I'll explain what a default route is in a second.

      For now, just assume that all data that you generate within your local network by default is sent through to your internet service provider.

      So now the packet that you've generated is inside your internet service provider on a router, and this router has multiple network interface cards connecting to all of those remote networks.

      Now let's assume in those remote networks is another router, and each of these routers uses the dot 1 IP address in each of those networks.

      So how does the ISP router inside Meow ISP know where to forward your data to?

      Well, it uses routes and route tables.

      Every router will have at least one route table.

      It could have more, which are attached to individual network interfaces, but for now let's keep things simple and assume that the router within our ISP has a single route table, and it will look something like this.

      A route table is a collection of routes.

      Each row in this table is an example route.

      It will have a destination field, and it will have a next hop or a target field.

      What happens is that every packet which arrives at this router, the router will check the packet's destination.

      What IP address is this packet destined for?

      And in this example, it's 52.217.13.37.

      Now at this point, the router will look for any routes in the route table which match the destination IP address of this packet.

      If multiple routes match, then it will prefer ones which are more specific.

      The two routes in yellows at the top and the bottom, these are examples of fairly specific routes.

      The one in blue in the middle is the inverse, this is not a specific route.

      The larger the prefix, so the higher the number after the slash, the more specific the route.

      So a slash 32 is the most specific, and a slash 0 is the least specific.

      A slash 32 actually represents one single IP address, and a slash 0, well this represents all IP addresses.

      A slash 24 means that the first 24 bits are for the network, and the last 8 bits are for the host.

      So this matches a network of 256 IP addresses.

      So for this packet that we have with the destination of 52.217.13.37, we've got two routes which match.

      The top route, which is 52.217.13.0/24, that network contains the IP address which our packet is destined for.

      So this matches.

      But also the middle route, 0.0.0/0, this matches, because this matches all IP addresses.

      The middle route is known as a default route.

      I mentioned before the packets from our home network on the left arrive at our ISP because there's a default route.

      Well this 0.0.0/0 is an example of a default route.

      This will match if nothing else does.

      Because we have two more specific routes in this route table, so the top and bottom, if either of those match, they will be selected rather than the default route in the middle.

      In this case the bottom route doesn't match our particular packet, only the top one matches.

      And so the top route will be selected because it's more specific than the default route.

      Now for the route that's selected, so the top route, it has a next hop or target field.

      This is the IP address which the packet is going to be forwarded to, to get one step closer through to its destination.

      Or in this case to arrive at the actual destination.

      And so the packet is forwarded through to this address.

      Routing as a process is where packets are forwarded or routed hop by hop across the internet from source to destination.

      Route tables are the thing which enables this.

      Route tables can be statically populated, or there are protocols such as BGP or the border gateway protocol, which allow routers to communicate with each other to exchange which networks they know about.

      And this is how the core of the internet functions.

      One important thing that you need to understand though, is that when our ISP router is forwarding the packet through to the AWS router, it's forwarding it at layer 2.

      It wraps the packet in a frame.

      The packet doesn't change.

      The frame though, it has the AWS routers MAC address as its destination.

      That's how the packet gets to the AWS router.

      But how do we determine the MAC address of the AWS router in this example?

      For that we use something called the address resolution protocol, and that's what I'm going to be covering next.

      This is the end of part 2 of this lesson.

      It's a pretty complex lesson, and so I wanted to give you the opportunity to take a small break, maybe stretch your legs, or make another coffee.

      Part 3 will continue immediately from this point, so go ahead, complete this video, and when you're ready, I look forward to you joining me in part 3.

    1. Welcome back.

      Now that we've covered the physical and data link layers, next we need to step through layer 3 of the OSI model, which is the network layer.

      As I mentioned in previous videos, each layer of the OSI model builds on the layers below it, so layer 3 requires one or more operational layer 2 networks to function.

      The job of layer 3 is to get data from one location to another.

      When you're watching this video, data is being moved from the server hosting the video through to your local device.

      When you access AWS or stream from Netflix, data is being moved across the internet, and it's layer 3 which handles this process of moving data from a source to a destination.

      To appreciate layer 3 fully, you have to understand why it's needed.

      So far in the series, I've used the example of 2-4 friends playing the game on a local area network.

      Now what if we extended this, so now we have 2 local area networks and they're located with some geographic separation.

      Let's say that one is on the east coast of the US and another is on the west coast, so there's a lot of distance between these 2 separate layer 2 networks.

      Now LAN1 and LAN2 are isolated layer 2 networks at this point.

      Devices on each local network can communicate with each other, but not outside of that local layer 2 network.

      Now you could pay for and provision a point-to-point link across the entire US to connect these 2 networks, but that would be expensive, and if every business who had multiple offices needed to use point-to-point links, it would be a huge mess and wouldn't be scalable.

      Additionally, each layer 2 network uses a shared layer 2 protocol.

      In the example so far, this has been Ethernet.

      Any networks where only using layer 2, if we want them to communicate with each other, they need to use the same layer 2 protocol to communicate with another layer 2 network.

      Now not everything uses the same layer 2 protocol, this presents challenges, because you can't simply join 2 layer 2 networks together, which use different layer 2 protocols and have them work out of the box.

      With the example which is on screen now, imagine if we had additional locations spread across the continental US.

      Now in between these locations, let's add some point-to-point links, so we've got links in pink which are tabled connections, and these go between these different locations.

      Now we also might have point-to-point links which use a different layer 2 protocol.

      In this example, let's say that we had a satellite connection between 2 of these locations.

      This is in blue, and this is a different layer 2 technology.

      Now Ethernet is one layer 2 technology which is generally used for local networks.

      It's the most popular wired connection technology for local area networks.

      But for point-to-point links and other long distance connections, you might also use things such as PPP, MPLS or ATM.

      Not all of these use frames with the same format, so we need something in common between them.

      Layer 2 is the layer of the OSI stack which moves frames, it moves frames from a local source to a local destination.

      So to move data between different local networks, which is known as inter-networking, this is where the name internet comes from.

      We need a layer 3.

      Layer 3 is this common protocol which can span multiple different layer 2 networks.

      Now layer 3 or the network layer can be added onto one or more layer 2 networks, and it adds a few capabilities.

      It adds the internet protocol or IP.

      You get IP addresses which are cross-networking addresses, which you can assign to devices, and these can be used to communicate across networks using routing.

      So the device that you're using right now, it has an IP address.

      The server which stores this video, it too has an IP address.

      And the internet protocol is being used to send requests from your local network across the internet to the server hosting this video, and then back again.

      IP packets are moved from source to destination across the internet through many intermediate networks.

      Devices called routers, which are layer 3 devices, move packets of data across different networks.

      They encapsulate a packet inside of an ethernet frame for that part of the journey over that local network.

      Now encapsulation just means that an IP packet is put inside an ethernet frame for that part of the journey.

      Then when it needs to be moved into a new network, that particular frame is removed, and a new one is added around the same packet, and it's moved onto the next local network.

      So as this video data is moving from my server to you, it's been wrapped up in frames.

      Those frames are stripped away, new frames are added, all while the packets of IP data move from my video server to you.

      So that's why IP is needed at a high level, to allow you to connect to all that remote networks, crossing intermediate networks on the way.

      Now over the coming lesson, I want to explain the various important parts of how layer 3 works.

      Specifically IP, which is the layer 3 protocol used on the internet.

      Now I'm going to start with the structure of packets, which are the data units used within the internet protocol, which is a layer 3 protocol.

      So let's take a look at that next.

      Now packets in many ways are similar to frames.

      It's the same basic concept.

      They contain some data to be moved, and they have a source and destination address.

      The difference is that with frames, both the source and destination are generally local.

      With an IP packet, the destination and source addresses could be on opposite sides of the planet.

      During their journey from source to destination packets remain the same, as they move across layer 2 networks.

      They're placed inside frames, which is known as encapsulation.

      The frame is specific to the local network that the packet is moving through, and changes every time the packet moves between networks.

      The packet though doesn't change.

      Normally it's constant for the duration for its entire trip between source and destination.

      Although there are some exceptions that I'll be detailing in a different lesson, when I talk about things like network address translation.

      Now there are two versions of the internet protocol in use.

      Version 4, which has been used for decades, and version 6, which adds more scalability.

      And I'll be covering version 6 and its differences in a separate lesson.

      An IP packet contains various different fields, much like frames that we discussed in an earlier video.

      At this level there are a few important things within an IP packet which you need to understand, and some which are less important.

      Now let's just skip past the less relevant ones.

      I'm not saying any of these are unimportant, but you don't need to know exactly what they do at this introductory level.

      Things which are important though, every packet has a source and destination IP address field.

      The source IP address is generally the device IP which generates the packet, and the destination IP address is the intended destination IP for the packet.

      In the previous example we have two networks, one east coast and one west coast.

      The source might be a west coast PC, and the destination might be a laptop within the east coast network.

      But crucially these are both IP addresses.

      There's also the protocol field, and this is important because IP is layer 3.

      It generally contains data provided by another layer, a layer 4 protocol, and it's this field which stores which protocol is used.

      So examples of protocols which this might reference are things like ICMP, TCP or UDP.

      If you're storing TCP data inside a packet this value will be 6, for PINs known as ICMP this value will be 1, and if you're using UDP as a layer 4 protocol then this value will be 17.

      This field means that the network stack at the destination, specifically the layer 3 component of that stack, will know which layer 4 protocol to pass the data into.

      Now the bulk of the space within a packet is taken up with the data itself, something that's generally provided from a layer 4 protocol.

      Now lastly there's a field called time to live or TTL.

      Remember the packets will move through many different intermediate networks between the source and the destination, and this is a value which defines how many hops the packet can move through.

      It's used to stop packets looping around forever.

      If for some reason they can't reach their destination then this defines a maximum number of hops that the packet can take before being discarded.

      So just in summary a packet contains some data which it carries generally for layer 4 protocols.

      It has a source and destination IP address, the IP protocol implementation which is on routers moves packets between all the networks from source to destination, and it's these fields which are used to perform that process.

      As packets move through each intermediate layer 2 network, it will be inserted or encapsulated in a layer 2 frame, specific for that network.

      A single packet might exist inside tens of different frames throughout its route to its destination, one for every layer 2 network or layer 2 point to point link which it moves through.

      Now IP version 6 from a packet structure is very similar, we also have some fields which matter less at this stage.

      They are functional but to understand things at this level it's not essential to talk about these particular fields.

      And just as with IP version 4, IP version 6 packets also have both source and destination IP address fields.

      But these are bigger IP version 6 addresses are bigger which means there are more possible IP version 6 addresses.

      And I'm going to be covering IP version 6 in detail in another lesson.

      It means though that space taken in a packet to store IP version 6 source and destination addresses is larger.

      Now you still have data within an IP version 6 packet and this is also generally from a layer 4 protocol.

      Now strictly speaking if this were to scale then this would be off the bottom of the screen, but let's just keep things simple.

      We also have a similar field to the time to live value within IP version 4 packets, which in IP version 6 this is called the hop limit.

      Functionally these are similar, it controls the maximum number of hops that the packet can go through before being discarded.

      So these are IP packets, generally they store data from layer 4 and they themselves are stored in one or more layer 2 frames as they move around networks or links which fall on the internet.

      Okay so this is the end of part 1 of this lesson.

      It was getting a little bit on the long side and I wanted to give you the opportunity to take a small break, maybe stretch your legs or make a coffee.

      Now part 2 will continue immediately from this point, so go ahead complete this video and when you're ready I look forward to you joining me in part 2.

    1. Welcome back and in this part of the lesson series I'm going to be discussing layer one of the seven layer OSI model which is the physical layer.

      Imagine a situation where you have two devices in your home let's say two laptops and you want to play a local area network or LAN game between those two laptops.

      To do this you would either connect them both to the same Wi-Fi network or you'd use a physical networking cable and to keep things simple in this lesson I'm going to use the example of a physical connection between these two laptops so both laptops have a network interface card and they're connected using a network cable.

      Now for this part of the lesson series we're just going to focus on layer one which is the physical layer.

      So what does connecting this network cable to both of these devices give us?

      Well we're going to assume it's a copper network cable so it gives us a point-to-point electrical shared medium between these two devices so it's a piece of cable that can be used to transmit electrical signals between these two network interface cards.

      Now physical medium can be copper in which case it uses electrical signals it can be fiber in which case it uses light or it can be Wi-Fi in which case it uses radio frequencies.

      Whatever type of medium is used it needs a way of being able to carry unstructured information and so we define layer one or physical layer standards which are also known as specifications and these define how to transmit and receive raw bitstream so ones and zeros between a device and a shared physical medium in this case the piece of copper networking cable between our two laptops so the standard defines things like voltage levels, timings, data rates, distances which can be used, the method of modulation and even the connector type on each end of the physical cable.

      The specification means that both laptops have a shared understanding of the physical medium so the cable.

      Both can use this physical medium to send and receive raw data.

      For copper cable electrical signals are used so a certain voltage is defined as binary 1 say 1 volt and a certain voltage as binary 0 say -1 volt.

      If both network cards in both laptops agree because they use the same standard then it means that zeros and ones can be transmitted onto the medium by the left laptop and received from the medium by the right laptop and this is how two networking devices or more specifically two network interface cards can communicate at layer one.

      If I refer to a device as layer X so for example layer one or layer three then it means that the device contains functionality for that layer and below so a layer one device just understands layer one and a layer three device has layers one, two and three capability.

      Now try to remember that because it's going to make much of what's coming over the remaining videos of this series much easier to understand.

      So just to reiterate what we know to this point we've taken two laptops we've got two layer one network interfaces and we've connected them using a copper cable a copper shared medium and because we're using a layer one standard it means that both of these cards can understand the specific way that binary zeros and ones are transmitted onto the shared medium.

      Now on the previous screen I use the example of two devices so two laptops with network interface cards communicating with each other.

      Two devices can use a point-to-point layer one link a fancy way of talking about a network cable but what if we need to add more devices a two-player game isn't satisfactory we need to add two more players for a total of four.

      Well we can't really connect these four devices to a network cable with only two connectors but what we can do is to add a networking device called a hub in this example it's a four-port hub and the laptop on the left and right instead of being connected to each other directly and now connected to two ports of that hub because it's a four-port hub this also means that it has two ports free and so it can accommodate the top and bottom laptops.

      Now hubs have one job anything which the hub receives on any of its ports is retransmitted to all of the other ports including any errors or collisions.

      Conceptually a hub creates a four connector network cable one single piece of physical medium which four devices can be connected to.

      Now there are a few things that you really need to understand at this stage about layer one networking.

      First there are no individual device addresses at layer one one laptop cannot address traffic directly at another it's a broadcast medium the network card on the device on the left transmits onto the physical medium and everything else receives it it's like shouting into a room with three other people and not using any names.

      Now this is a limitation but it is fixed by layer two which will cover soon in this lesson series.

      The other consideration is that it is possible that two devices might try and transmit at once and if that happens there will be a collision this corrupts any transmissions on the shared medium only one thing can transmit at once on a shared medium and be legible to everything else if multiple things transmit on the same layer one physical medium then collisions occur and render all of the information useless.

      Now related to this layer one has no media access control so no method of controlling which devices can transmit so if you decide to use a layer one architecture so a hub and all of the devices which is shown on screen now then collisions are almost guaranteed and the likelihood increases the more layer one devices are present on the same layer one network.

      Layer one is also not able to detect when collisions occur remember these network cards are just transmitting via voltage changes on the shared medium it's not digital they can in theory all transmit at the same time and physically that's okay it means that nobody will be able to understand anything but at layer one it can happen so layer one is done it doesn't have any intelligence beyond defining the standards that all of the devices will use to transmit onto the shared medium and receive from the shared medium because of how layer one works and because of how a hub works because it simply retransmits everything even collisions then the layer one network is said to have one broadcast and one collision domain and this means that layer one networks tend not to scale very well the more devices are added to a layer one network the higher the chance of collisions and data corruption.

      Now layer one is fundamental to networking because it's how devices actually communicate at a physical level but for layer one to be useful for it to be able to be used practically for anything else then we need to add layer two and layer two runs over the top of a working layer one connection and that's what we'll be looking at in the next part of this lesson series.

      As a summary of the position that we're in right now assuming that we have only layer one networking we know that layer one focuses on the physical shared medium and it focuses on the standards for transmitting onto the medium and receiving from the shared medium so all devices which are part of the same layer one network need to be using the same layer one medium and device standards generally this means a certain type of network card and a certain type of cable or it means why vicar's using a certain type of antennas and frequency ranges what layer one doesn't provide is any form of access control of the shared medium and it doesn't give us uniquely identifiable devices and this means we have no method for device to device communication everything is broadcast using transmission onto the shared physical medium.

      Now in the next video of this series I'm going to be stepping through layer two which is the data link layer and this is the layer which adds a lot of intelligence on top of layer one and allows device to device communication and it's layer two which is used by all of the upper layers of the OSI model to allow effective communication but it's important that you understand how layer one works because this physically is how data moves between all devices and so you need to have a good fundamental understanding of layer one.

      Now this seems like a great place to take a break so I'm going to end this video here so go ahead and complete this video and then when you're ready I look forward to you joining me in the next part of this lesson series where we'll be looking at layer two or the data link layer.

    1. Welcome back, this is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      Now the only thing that remains is just to test out this configuration.

      And to do that we're going to launch an EC2 instance into the WebA subnet.

      So click on services and just type EC2 to move across to the EC2 console.

      Now once we're on the EC2 console, just click on launch instance.

      Then you'll be taken to the launch instance console.

      Into the name box, just go ahead and type a4l-bastian.

      Scroll down and we're going to create a bastion instance using Amazon Linux.

      So click on Amazon Linux.

      In the dropdown below, go ahead and select the latest version of Amazon Linux.

      Just make sure that it does say free tier eligible on the right of this dropdown.

      Assuming that's all good, just below that make sure that in the architecture dropdown it's set to 64-bit x86.

      Moving down further still, under instance type, just make sure that this is set to a free tier eligible instance.

      It should default to T2.micro or T3.micro.

      Depending on your region, either of these could be free tier eligible.

      In my case it's T2.micro, but whatever your shows, just make sure that it's similar sized and says free tier eligible.

      Now directly below that, under key pair, just click in this box.

      You should at this point in the course have a key pair creator called a4l.

      If you do, go ahead and select that key pair in the box.

      If you don't, don't worry, you can just go ahead and click on create new key pair.

      Enter a4l into the key pair name, select RSA, and then select PEM for the private key format and click on create key pair.

      This will download the key pair to your local machine and then you can continue following along with this video.

      So select that from the dropdown.

      Directly below, under network settings click on edit.

      This instance is going to go into the animals for live VPC.

      So click on the VPC dropdown and select a4l-vpc1.

      Directly below that, click in the subnet dropdown and we want to go ahead and look for sn-web-a.

      So select the weba subnet.

      This should change both of the dropdowns below.

      So auto assign public IP and auto assign IPv6 IP to enable.

      So just make sure that both of those are set to enable.

      Directly below this, make sure that create security group is checked.

      We're going to create a new security group.

      Under security group name, just go ahead and enter a4l-bassian-sg and then put that same text in the description box directly below.

      Now all of these defaults should be good.

      Just make sure it's set to SSH, source anywhere.

      Make sure that 0.0.0.0/0 and double colon 4/0 are both present directly below source.

      Everything else looks good.

      We can accept the rest of the defaults.

      Just go ahead and click on launch instance.

      Then click on instances at the top left of the screen.

      At this point the instance is launching and we'll see that a4l-bassian is currently running.

      We'll see the status check is showing initializing.

      So we need to give this instance a few minutes to fully provision.

      So go ahead and pause this video and we're going to resume it once this instance is ready to go.

      And it has two out of two status checks.

      So our instance is now showing two out of two status checks.

      And that means everything's good and we're ready to connect.

      Now if you select the instance, you'll see in the details pane below how it has a public IP version 4 address, a private IP version 4 address, public and private IP version 4 DNS.

      And if we scroll down, you'll see lots of other information about this instance.

      Now we're only concerned with the public IP version 4 address.

      We're going to go ahead and connect to this instance this time using a local SSH client on our machine.

      So right click and then select connect.

      Now if we want to quickly connect into this instance, we can choose to use EC2 instance connect, which is a way to connect into the instance using a web console.

      Now this does need an instance with a public IP version 4 address, but we have allocated a public address.

      So if we wanted to, we can just make sure that the username is correct.

      It should be EC2-user.

      If we hit connect, it will open up a connection to this instance using a web console.

      And this is often much easier to connect to EC2 instances if you don't have access to a local SSH client, or if you just want to quickly connect to perform some administration.

      We can also connect with an SSH client.

      If we select SSH client, it gives us the commands to run in order to connect to this EC2 instance.

      So right at the bottom is an example connect command.

      So SSH, we pick the key to use and then we pick the user at and then the public IP version 4 DNS.

      So if we copy that into our clipboard and then move across to our terminal or command prompt, move into the folder where you downloaded the SSH keybearant to, in my case downloads, and paste in that command and press enter, that should connect us to the EC2 instance.

      We'll have to verify the fingerprint, so we need to verify the authenticity of this host.

      For this purpose, we can just go ahead and answer yes and press enter.

      Now if it's the first time we're connecting using a particular key, and if you're running either macOS or Linux, you might be informed that the permissions on the key are too open.

      In this case, the permissions are 0, 6, 4, 4, which are too open and we get this error.

      Now it's possible to correct that if we move back to the AWS console.

      It also gives us the command to correct these permissions.

      So CHmod, space 400, space and then the name of the key.

      So I'm going to copy that into my clipboard and move back to my terminal, paste that in and press enter, and that will correct those permissions.

      Now if I get the connection command again, so copy that into my clipboard, and this time I'll paste it in and press enter and now I will be connected to this EC2 instance.

      Now if you're doing this demonstration on Windows 10, you probably won't have to correct those permissions.

      This is something specific to macOS or Linux.

      So whenever you're connecting to EC2 instances which have a public IP version 4 address, you've always got the ability to use either EC2 instance connect or a local SSH client.

      Now the third option which is session manager, this is a way that you can connect to instances even if they don't have public IP version 4 addressing.

      And I'll be detailing this product fully later on in the course because there is some additional configuration that's required.

      Now this bastion host, it's an EC2 instance and it does fall under the free tier.

      So because it's a T2.micro or whatever type of instance you picked which falls under the free tier, you're not going to be billed for any usage of this instance in a given month.

      Now as a general rule, as you're moving through the course, if you're ever intending to take a break, then you always have the option of deleting all of the infrastructure that you've created within a specific demo lesson.

      So most of the more complex demo lessons that you'll have moving through the course, at the end of every demo lesson there will be a brief set of steps where I explain how to clean up the account and return it into the same state as it was at the start of the lesson.

      But in certain situations I might tell you that one option is not to delete the infrastructure.

      Whether you do delete it or not depends on whether you're intending to complete the next demo straight away or whether you're taking a break.

      Now in this particular case I'm going to demonstrate exactly how you can clear up this infrastructure. [background noise] [background noise] In the next demo lesson you're going to be continuing using this structure, but I'm going to demonstrate how you can automate the creation using a CloudFormation template.

      To clear up this infrastructure though, go ahead, right click on this bastion host and select terminate instance.

      You'll need to click terminate to confirm and that will terminate and delete the instance.

      You won't be charged for any further usage of that instance.

      We need to wait for that instance to fully terminate, so pause the video and wait for it to move into a terminated state and then we can continue.

      So that instance is terminated and now that that's done, we can click on services and move across to the VPC console and we're going to delete the entire Animals for Life VPC.

      And don't worry, in the next demo lesson I'll explain how we can automate the creation.

      So for now and in the course we're going to be using much more automation so that anything that you've done previously, we're going to automate the creation and focus your valuable time only on the things that you've just learned.

      So click on your VPCs.

      It should list two VPCs, the default one and the Animals for Life VPC.

      Select the Animals for Life VPC, click on Actions and then delete the VPC.

      Now this is going to delete all of the resources that are associated with this VPC.

      So the internet gateway, the route tables and all of the subnets that you've created as part of the demo lessons to this point in the course.

      So go ahead and type delete and then click delete to confirm that process and that will fully tidy up the account and return it into the same state as it was at the start of the VPC section.

      Now with that being said, this is the end of this lesson.

      You've successfully converted three subnets, so Web A, Web B and Web C to be public and you've done that by creating an internet gateway, associating that with the VPC, creating a route table, associating that with those subnets, adding two routes, pointing those routes at the internet gateway and then configuring those subnets to allocate a public IP version for address to any resources launched into those subnets.

      So that's the same set of steps that you'll need to do to make any subnets public from an IP version for perspective in future.

      So this is going to be the same tasks that you would use in larger production projects.

      Although in production, you would probably automate it and don't just show you how to do that as you move through the course.

      Now at this point, you've finished everything that you need to do in this demo lesson, so great job.

      You've actually created something that is production ready and production useful.

      Over the remainder of this section of the course, we're going to refine the design that we've got and add additional capabilities.

      So in the upcoming lessons, I'll be talking about network address translation and how that can be used to give private EC2 instances access to the internet for things like software updates.

      We'll be talking about the security of subnets using network access control lists known as knuckles and much, much more.

      But you're doing a fantastic job so far.

      This is not a trivial thing that you've implemented to this point.

      So really great job.

      But at this point, just go ahead and complete the video.

      And then when you're ready, I look forward to you joining me in the next. [no audio]

    1. Welcome back.

      This demo is going to bring together some really important theory and architecture that you've learned over the past few lessons.

      What we're starting this demo lesson with is this architecture.

      We have our VPC, the Animals for Life VPC in US East 1.

      It uses the 10.16.0.0/16 side range.

      It has 12 subnets created inside it, over three AZs with four tiers, Reserve, DB, Application and Web.

      Now currently all the subnets are private and can't be used for communication with the internet or the AWS public zone.

      In this demo we want to reconfigure the VPC to allow that.

      So the first step is to create an internet gateway and attach it.

      To do that, I'm going to move across to my desktop.

      Now to do this in your environment, you'll need the VPC and subnet configuration as you set it up in the previous demo lesson.

      So that configuration needs to be in place already.

      You need to be logged in as the I am admin user of the management account of the organization and have the Northern Virginia region selected, so US - East - 1.

      So go ahead and move across to the VPC console.

      Now this should already be in the recently visited services because you were using this in the previous demo lesson, but if it's not visible just click in the services drop down, type VPC and then click to move to the VPC console.

      Now if you do still have the configuration as it was at the end of the previous demo lesson, you should be able to click on subnets on the menu on the left and see a list of lots of subnets.

      You'll see the ones for the default VPC without a name.

      And if you have the correct configuration, you should see a collection of 12 subnets, 3 application subnets, 3 database subnets, 3 reserved subnets and then 3 web subnets.

      So all of these should be in place within the Animals for Life VPC in order to do the tasks within this demo lesson.

      So I'm going to assume from this point onwards that you do have all of these subnets created and configured.

      Now what we're going to be doing in this demo lesson is configuring the 3 web subnets, so web A, web B and web C, to be public subnets.

      Being a public subnet means that you can launch resources into the subnet, have them allocated with a public IP version 4 address and have connectivity to and from the public IP version 4 internet.

      And in order to enable that functionality, there are a number of steps that we need to perform and I want you to get the practical experience of implementing these within your own environment.

      Now the first step to making subnets public is that we need an internet gateway attached to this VPC.

      So internet gateways, as I talked about in the previous theory lesson, are highly available gateway objects which can be used to allow public routing to and from the internet.

      So we need to create one.

      So let's click on internet gateways on the menu on the left.

      They'll already be an internet gateway in place for the default VPC.

      Remember when you created the default VPC, all this networking infrastructure is created and configured on your behalf.

      But because we created a custom VPC for animals for life, we need to do this manually.

      So to do that, go ahead and click on create internet gateway.

      We're going to call the internet gateway A4L, so animals for life, VPC 1, which is the VPC we're going to attach it to and then IGW for internet gateway.

      So A4L-VPC1-IGW.

      Now that's the only information that we need to enter, so scroll down and click on create internet gateway.

      Internet gateways are initially not attached to a VPC and we can tell that because it's initially in.

      We need to attach this to the animals for life VPC.

      So click on actions and then attach to VPC inside the available VPCs box.

      Just click and then select A4L-IVAN-VPC1.

      Once selected, go ahead and click on attach internet gateway and that will attach our brand new internet gateway to the animals for life VPC.

      And that means that it's now available within that VPC as a gateway object, which gives the VPC the capability to communicate to and receive communications from the public internet and the AWS public zone.

      Now the next step is that we want to make all the subnets in the web tier public, so the services deployed into these subnets can take advantage of this functionality.

      So we want the web subnets to be able to communicate to and receive communications from the public internet and AWS public services.

      Now there are a number of steps that we need to do to accomplish this.

      We need to create a route table for the public subnets.

      We need to associate this route table with the three public subnets, so web A, web B and web C and then we need to add two routes to this route table.

      One route will be a default route for IP version 4 traffic and the other will be a default route for IP version 6 traffic.

      And both of these routes for their target will be pointing at the internet gateway that you've just created and attached to this VPC.

      Now this will configure the VPC router to forward any data intended for anything not within our VPC to the internet gateway.

      Finally, on each of these web subnets will be configuring the subnet to auto assign a public IP version 4 address and that will complete the process of making them public.

      So let's perform all of these sets of configuration.

      So now that we're back at the AWS console, we need to create a route table.

      So go ahead and click on route tables on the menu on the left and then we're going to create a new route table.

      First we'll select the VPC that this route table will belong to and it's going to be the animals for life.

      If a VPC, so go ahead and select that VPC and then we're going to give this route table a name.

      And I like to keep the naming scheme consistent, so we're going to use A4, L4, animals for life and then a hyphen.

      VPC1 because this is the VPC the route table will belong to and then hyphen RT for route table and then hyphen and then web because this route table is going to be used for the web subnets.

      So go ahead and create this route table and click route tables on the menu on the left.

      If we select the route table that we've just created, so that's the one that's called A4, L hyphen VPC1 hyphen RT hyphen web and then just expand this overview area at the bottom.

      We'll be able to see all the information about this route table.

      Now there are a number of areas of this which are important to understand.

      One is the routes area which lists all the routes on this route table and the other is subnet associations.

      This determines which subnets this route table is.

      So let's go to subnet associations and currently we can see that it's not actually associated with any subnets within this VPC.

      We need to adjust that so go ahead and edit those associations and we're going to associate it with the three web subnets.

      So you need to select web A, web B and web C.

      Now notice how all those are currently associated with the main route table of the VPC.

      Remember a subnet can only be associated with one route table at a time.

      If you don't explicitly associate a route table with a subnet then it's associated with the main route table.

      We're going to change that.

      We're going to explicitly associate this new route table with the web A, web B and web C subnets.

      So go ahead and say that.

      So now this route table has been associated with web A, web B and web C.

      Those subnets are no longer associated with the main route table of the VPC.

      So now we've configured the association as a route to routes.

      And we can see that this route table has two local routes.

      We've got the IP version 4 side of the VPC and the app.

      IP version 6 side of the VPC.

      So these two routes on this route table will mean that web A, web B and web C will know how to direct traffic towards any other IP version 4 or IP version 6 addresses within this VPC.

      Now these local routes can never be adjusted or removed, but what we can do is add additional routes.

      So we're going to add two routes, a default route for IP version 4 and a default route for IP version 6.

      So we'll do that.

      We'll start with IP version 4.

      So we'll edit those routes and then we'll add a route.

      The format for the IP version 4 default route is 0.0.0.0/0.

      And this means any IP addresses.

      Now I've talked elsewhere in the course how there is a priority to routing.

      Within a VPC there's a more specific route always takes priority.

      So this route, the /16 is more specific than this default route.

      So this default route will only affect IP version 4 traffic, which is not matched by this local route.

      So essentially anything which is IP version 4, which is not destined for the VPC, will use this default route.

      Now we need to pick the internet gateway as the target for this route.

      So click in the target box on this row, select internet gateway.

      There should only be one that's highlighted.

      That's the Animals for Life internet gateway you created moments ago.

      So select that and that means that any IP version 4 traffic which is not destined for the VPC side of it. [siren] Range will be sent to the Internet Gateway.

      Now we're going to do the same for IPv6.

      So go ahead and add another route.

      And the format for IPv60 default routes is double colon, forward slash zero.

      And this is the same architecture. [siren] It essentially means this matches all IPv6 addresses, but it's less specific than the IPv6 and version 6 local route on this top row.

      So this will only be used for any IPv6 addresses which are not in the IPv6 VPC side range.

      So go ahead and select Target, go to Internet Gateway, and select the Animals for Life Internet Gateway.

      And once you've done both of those, go ahead and click on Save Changes.

      Now this means that we now have two default routes, an IPv4 default route, and an IPv6 default route.

      So this means that anything which is associated with these route tables will now send any unknown traffic towards the Internet Gateway.

      But what we need to do before this works is we need to ensure that any resources launched into the Web A, Web B, or Web C subnets are allocated with public IPv4 addresses.

      To do that, go ahead and click on Subnets.

      In the list, we need to locate Web A, Web B, and Web C.

      So we'll start with Web A, so select Web A, click on Actions, and then Edit Subnet Settings.

      And this time, we're going to modify this subnet so that it automatically assigns a public IPv4 address.

      So check this box into the Save, and that means that any resources launched into the Web A subnet will be allocated with a public IPv4 address.

      Now we need to follow the same process for the other web subnets, so select the Web B subnet, click on Actions, and then Edit Subnet Settings.

      Enable IPv4, click on Save, and then do that same process for Web C.

      So locate Web C, click on Actions, and then Edit Subnet Settings.

      And then enable public IPv4 addresses and click on Save.

      So that's all the network configuration done.

      We've created an Internet Gateway.

      We've associated the Internet Gateway with the Animals for IPPC.

      We've created a Routetable for the web subnets.

      We've associated this Routetable with the web subnets.

      We've added default routes onto this Routetable, pointing at the Internet Gateway as a default IPv4 and IPv6 route.

      And then we've enabled the allocation of public IPv4 addresses for Web A, Web B, and Web C.

      Okay, so this is the end of Part 1 of this lesson.

      It was getting a little bit on the long side, and so I wanted to add a break.

      It's an opportunity just to take a rest or grab a coffee.

      Part 2 will be continuing immediately from the end of Part 1.

      To go ahead, complete the video, I'm ready, join me in part two.

    1. Welcome back.

      And in this lesson, I want to talk about how routing works within a VPC and introduce the internet gateway, which is how we configure a VPC so that data can exit to and enter from the AWS public zone and public internet.

      Now, this lesson will be theory where I'm going to introduce routing and the internet gateway to the architecture behind both those things, as well as jump boxes also known as Bastion hosts.

      In the demo lesson, which immediately follows this one, you'll get the opportunity to implement an internet gateway yourself and the animals for life at VPC and fully configure the VPC with public subnet that allow you to connect to that jump box.

      So let's get started.

      We've got a lot to cover.

      A VPC router is a highly available device which is present in every VPC, both default or custom, which moves traffic from somewhere to somewhere else.

      It runs in all the availability zones that the VPC uses and never needs a way about its availability.

      It simply works.

      The router can be networked in every subnet, and the network is just one address of the subnet.

      By default in a custom VPC, without any other configuration, the VPC router will simply route traffic between subnets in that VPC.

      If an EC2 instance in one subnet wants to communicate with something in another subnet, the VPC router is the thing that moves the traffic between subnets.

      Now, the VPC router is controllable.

      You create route tables which influence what's to do with traffic when it leaves a subnet.

      So just to be clear, the route table that's associated with a subnet defines what the VPC router will do when data leaves that subnet.

      A VPC is created with what's known as a main route table.

      If you don't explicitly associate a custom route table with a subnet, it uses the main route table of the VPC.

      If you do associate a route table that you create with a subnet, then when you associate that, the main route table is disassociated.

      A subnet can only have one route table associated with it at any one time, but a route table can be associated with many subnets.

      A route table looks like this in the user interface.

      In this case, this is the main route table for this specific VPC.

      And a route table is just a list of routes.

      This is one of those routes.

      When traffic leaves the subnet that this route table is associated with, the VPC router reviews the IP packets.

      And remember, I said that a packet had a source address and a destination address, as well as some data.

      The VPC router looks at the destination address of all packets leaving the subnet.

      And when it has that address, it looks at the route table and it identifies all the routes which match that destination address.

      And it does that by checking the destination field at the route.

      This destination field determines what destination the route matches.

      Now, the destination field on a route could match exactly one specific IP address.

      It could be an IP with a /32 prefix.

      And remember, that means that it matches one single IP.

      But the destination field on a route could also be a network match.

      So matching an entire network of which that IP is part.

      Or it could be a default route.

      Remember, for IP version 4, I mentioned that 0.0.0.0.0/0 matches all IP version 4 IP addresses.

      That's known as a default route, a catchall.

      In the case where traffic leaving a subnet only matches one route, then that one route is selected.

      If multiple routes match, so maybe there's a specific /32 IP match, maybe there's a /16 network match, and maybe there's a 0.0.0.0/0 default match, well then the prefix is used as a priority.

      The higher the prefix value, the more specific the route is and the higher priority that that route has.

      So the higher the prefix, all the way up to the highest priority of /32, that is used to select which route applies when traffic leaves a subnet.

      Once we get to the point where a single rule in a route table is selected, either the sole route that applies or the one with the highest priority, then the VPC router forwards that traffic through to its destination, which is determined by the target field on the route.

      And the target field will either point at an AWS gateway, or it will say, as with this example, local.

      And local means that the destination is in the VPC itself, so the VPC router can forward the traffic directly.

      All route tables have at least one route, the local route.

      This matches the VPC side range, and it lets the VPC router know that traffic destined for any IP address in the VPC side range is local and it can be delivered directly.

      If the VPC is also IPv6 enabled, people also have another local route matching the IPv6 side for the VPC.

      As is the case with this example, that bottom route, beginning 2600, that is an IPv6 local route.

      That's the IPv6 side of this specific VPC.

      Now, these local routes can never be updated.

      They're always present, and the local routes always take priority.

      They're the exception to that previous rule about the more specific the route is, the higher the priority.

      Local routes always take priority.

      For the exam, remember the route tables are attached to zero or more subnets.

      A subnet has to have a route table.

      It's either the main route table of the VPC or a custom one that you've created.

      A route table controls what happens to data as it leaves the subnet or subnets that that route table is associated with.

      Local routes are always there, uneditable, and match the VPC IPv4 or VPC side range.

      For anything else, higher, prefix values are more specific than they take priority.

      The way the route works is it matches a destination IP, and for that route, it directs traffic towards a specific target.

      Now, a default route, which I'll talk about shortly, is what happens if nothing else matches.

      Now, an internet gateway is one of the most important add-on features available within a VPC.

      It's a regionally resilient gateway which can be attached to a VPC.

      I've highlighted the words "region" and "resilience" because it always comes up in the exam.

      You do not need a gateway per availability zone.

      The internet gateway's resilient by design.

      One internet gateway will cover all of the availability zones in the region which the VPC is using.

      Now, there's a one-to-one relationship between internet gateways and the VPC.

      A VPC can have no internet gateways which makes it entirely private, or it can have one internet gateway.

      Those are the two choices.

      An internet gateway can be created and not attached to a VPC, so it can have zero attachments, but it can only ever be attached to one VPC at a time, at which point it's valid in all of the availability zones that the VPC uses.

      Now, the internet gateway runs from the border of the VPC and the AWS public zone.

      It's what allows services inside the VPC, which are allocated with public IP version 4 addresses or IP version 6 addresses, to be reached from the internet and to connect to the AWS public zone or the internet.

      Of course, the AWS public zone is used if you're accessing S3, SQS, SNS, or any other AWS public services from the VPC.

      Now, it's a managed gateway and so AWS handles the performance.

      From your perspective as an architect, it simply works.

      Now, using an internet gateway within a VPC, it's not all that complex.

      It's a simplified VPC diagram.

      First, we create and attach an internet gateway to a VPC.

      This means that it's available for use inside the VPC.

      We can use it as a target within route tables.

      So then we create a custom route table and it's within route tables.

      So then we create a custom route table and we associate it with a web subnet.

      Then we add IP version 4 and optionally IP version 6 default routes to the route table with the target being the internet gateway.

      Then finally, we configure the subnet to allocate IP version 4 addresses and optionally IP version 6 by default.

      And at that point, once we've done all of those actions together, the subnet is classified as being a public subnet and any services inside that subnet with public IP addresses can communicate to the internet and vice versa and they can communicate with the AWS public zone as long as there's no other security limitations that are in play.

      Now, don't worry if this seems complex.

      You'll get to experience it shortly in the upcoming demo lesson.

      But before that, I want to talk about how IP version 4 addressing actually works inside the VPC because I've seen quite a few difficult questions on the exam based around IP version 4 addressing and I want to clarify exactly how it works.

      So conceptually, this is how an EC2 instance might look if it's using IP version 4 to communicate with a software update server of some kind.

      So we've got the instance on the left with an internet gateway in between and let's say it's a Linux EC2 instance trying to do some software updates to a Linux update server that's located somewhere in the public internet zone.

      So the instance has a private IP address of let's say 10.16.16.20 and it also has an IP version 4 public address that's assigned to it of 43.250.192.20.

      Only that's not how it really works.

      This is another one of those little details which I try to include in my training courses because it really comes invaluable for the exam.

      What actually happens with public IP version 4 addresses is that they never touch the actual services inside the VPC.

      Instead, when you allocate a public IP version 4 address, for example, to this EC2 instance, a record is created which the internet gateway maintains.

      It links the instance's private IP to its allocated public IP.

      So the instance itself is not configured with that public IP.

      That's why when you make an EC2 instance and allocate it a public IP version 4 address, inside the operating system, it only sees the private IP address.

      Keep this in mind for the exam that there are questions which will try to trip you up on this one.

      For IP version 4, it is not configured in the OS with the public IP address.

      So let's look at the flow of data.

      How does this work?

      Well, when the Linux instance wants to communicate with the Linux software update server, it creates a packet of data.

      Now obviously it probably creates a lot of packets.

      Well, let's focus on one for now because it keeps the diagram nice and simple.

      The packet has a source address of the EC2 instance and a destination address of the Linux software update server.

      So at this point, the packet is not configured with any public addressing.

      This packet would not be routable across the public internet.

      It could not reach the Linux update server.

      That's really important to realize.

      Now the packet leaves the instance and because we've configured a default route, it arrives at the internet gateway.

      The internet gateway sees that this packet is from the EC2 instance because it analyzes the source IP address of that packet.

      And it knows that this instance has an associated public IP version 4 address.

      And so it adjusts the packet.

      It changes the packet's source IP address to the public IP address that's allocated to that instance.

      And this IP address, because it's public, is routable across the internet.

      So the internet gateway then forwards the updated packet onto its destination.

      So as far as the Linux software update server is concerned, it's receiving a packet from a source IP address of 43.250.192.20.

      It knows nothing of the private IP address of the EC2 instance.

      Now on the way back, the inverse happens.

      The Linux software update server wants to send a packet back to our EC2 instance.

      But as far as it's concerned, it doesn't know about the private address.

      It just knows about the public address.

      So the software update server sends a packet back, addressed to the instance's public IP address with its source address.

      So it thinks that the real IP address of the instance is this 43.250.192.20 address.

      Now this IP address actually belongs to the internet gateway.

      And so it travels over the public internet and it arrives at the internet gateway, which then modifies this packet.

      It changes it.

      It changes the destination address of this packet from the 43 address to the original IP address of the EC2 instance.

      And it does this because it's got a record of the relationship between the private IP and the allocated public IP.

      So it just changes the destination to the private IP address of the instance.

      And then it forwards this packet through the VPC network to the original EC2 instance.

      So the reason I wanted to highlight this is because at no point is the operating system on the EC2 instance aware of its public IP.

      It just has a private IP.

      Don't fall for any exam questions which try to convince you to assign the public IP version 4 address of an EC2 instance directly to the operating system.

      It has no knowledge of this public address.

      Configuring an EC2 instance appropriately using IP version 4 means putting the private IP address only.

      The public address never touches the instance.

      For IP version 6, all addresses that AWS users are natively, publicly, routable.

      And so in the case of IP version 6, the operating system does have the IP address version 6 address configured upon it.

      That's the publicly routable address.

      And all the internet gateway does is pass traffic from an instance to an internet server.

      And then back again, it doesn't do any translation.

      Now, before we implement this in the demo lesson, I just want to briefly touch upon bastion hosts and jump boxes.

      At a high level, bastion hosts and jump boxes are one and the same.

      Essentially, it's just an instance in a public subnet inside of VPC.

      And architecturally, they're used to allow incoming management connections.

      So all incoming management connections arrive at bastion hosts or jump box.

      And then once connected, you can then go on to access internal only VPC resources.

      So bastion hosts and jump boxes are generally used either as a management point or as an entry point for private only VPCs.

      So if your VPC is a highly secure private VPC, you'll generally have a bastion host or a jump box being the only way to get access to that VPC.

      So it's essentially just an inbound management point.

      And you can configure these bastion hosts or jump boxes to only accept connections from certain IP addresses, to authenticate with SSH, or to integrate with your on-premises identity servers.

      You can configure them exactly how you need, but at a real core architectural level, they are generally the only entry point to a highly secure VPC.

      And historically, they were the only way to manage private VPC instances.

      Now, there are alternative ways to do that now, but you will still find bastion hosts and jump boxes do feature on the exam.

      OK, so that's all the theory that I wanted to cover in this lesson.

      It's now time for a demo.

      In the next lesson, we're going to implement the Internet gateway in the Animals for Life VPC.

      We'll create it, we'll attach it to the VPC, we'll create a custom route table for the web subnets, we'll create two routes in that route table, one for IP version 4 and one for IP version 6.

      And both of these will point at the Internet gateway as a target.

      We'll associate that route table with the web tier subnets, configure those subnets to allocate public IP version 4 addresses, and then launch a bastion host into one of those web subnets.

      And if all goes well, we will be able to connect to that instance using our SSH application.

      So I think this demo lesson is going to be really interesting and really exciting.

      It's the first time that we're going to be stepping through something together that we could qualify as production like.

      Something that you could implement and would implement in a production ready VPC.

      So go ahead, complete this video, and when you're ready, join me in the demo lesson.

    1. Welcome back and in this demo lesson you're going to create all of the subnets within the custom VPC for animals for life.

      So we're going to create the subnets as shown on screen now.

      We're going to be creating four subnets in each availability zone.

      So that's the web subnet, the application subnet, the database subnet and the reserved subnet.

      And we're going to create each of those four in availability zone A, B and C.

      Now before we get started, attached to this lesson is a link to this document.

      Now this is a list of all the details of the subnet you're going to create in this demo lesson.

      So we've got a reserved subnet, a database subnet, an app subnet and a web subnet.

      And we've got each of those in availability zone A, B and C.

      Now in terms of what this document contains, we have a subnet name, then we have the IP range that subnet will use, the availability zone that subnet is within and then this last value and we'll talk about what this is very shortly.

      This relates to IP version 6.

      Now you'll notice that for each subnet this is a unique value.

      You're going to need this to configure the IP version 6 configuration for each of the subnets.

      Now let's get started.

      So let's move to the AWS console and you need to be within the VPC console.

      So if you're not already there, go ahead and type that in the search box at the top and then click to move to the VPC console and then click on subnets.

      And once you've done that, go ahead and click on create subnet.

      Now the newest version of the user interface allows you to create multiple subnets at a time and so we're going to create all four of the subnets in each availability zone.

      So we'll do this three times, one for availability zone A, one for B and one for C.

      So we'll get started with availability zone A and first we need to select the VPC to use.

      So click in the VPC ID drop down and select the animals for live VPC.

      Once you've done that, we're going to start with subnet one of one.

      So let's move to the subnet's document.

      So it's these subnets that we're going to create and we'll start with the reserved subnet.

      So copy the name of the subnet into your clipboard and paste it into subnet name.

      Then change the availability zone to AZA and then make sure IPv4 is set to manual and move back to the subnet's document and copy the IP range that we're going to use and paste that into this box.

      Then scroll down again and make sure manual input is selected for IPv6.

      Then click the drop down and select the IPv6 range for the VPC.

      Now the VPC uses a /56 IPv6 range.

      Because we need our subnet to fit inside this, we're going to make the individual subnet ranges much smaller.

      So what I'll need you to do and you'll need to do this each time is to click on the down arrow and you'll need to click on the down arrow twice.

      The first time will change it to a /60 and the second time to a /64.

      Now note in my case how I have 9, 6, 0, 0 and if I click on this right arrow, it increments this value by one each time.

      Now this value corresponds to the value in the subnet's document, so in this case 0, 0.

      By changing this value each time, it means that you're giving a unique IPv6 range to each subnet.

      So in this case, start off by leaving this set to 0, 0.

      And once you've done that, you can click on add new subnet.

      And we're going to create the next subnet.

      So in this case, it's sn-db-a, so enter that name, change the availability zone to A, manual input for IPv4, copy the IP range for DBA into your clipboard, paste that in, manual for IPv6, change the VPC range in the drop down, click the down arrow twice to set this to /64, and then change this value to 0, 1.

      And again, this matches the IPv6 value in the subnet's document.

      Then we'll do the same process for the third subnet, so we're going to add a new subnet.

      This time the name is sn-app-a, enter that, availability zone A, manual for IPv4, and then paste in the IP range, manual for IPv6, and select the VPC range, and then change the subnet block to /64 by clicking the down arrow twice.

      And then click the right arrow twice to set 0, 2 as the unique value for the subnet.

      And again, this matches the value in the subnet's document.

      Then lastly, we're going to do the same thing for the last subnet in availability zone A, so we're going to add a new subnet.

      This time it's web A, so copy and paste that into the subnet name box, set availability zone A, manual for IPv4, copy and paste the range from the subnet's document, manual for IPv6, select the IPv6 range from the VPC, click the down arrow twice to set the appropriate size for the subnet IPv6 range, and then click on the right arrow to change this value to 0, 3.

      Now that's all the subnet's created, all four of them in availability zone A, so we can scroll all the way down to the bottom, and click on create subnet, and that's going to create all four of those subnets.

      We can see those in this list.

      Now we're going to follow that same process for availability zone B, so click on create subnet, change the VPC to the animals for IPvc, and now we're going to start moving through quicker.

      So first we're going to do the reserve B subnet, so copy the name, paste that in, set the availability zone this time to B, manual for IPv4, paste in the range, manual for IPv6, select the VPC range in the drop-down, click on the down arrow twice to set the /64, and then click on the right arrow and set to 0, 4 as the unique value.

      Scroll down, click add new subnet, next is DBB, enter that, availability zone B, manual for IPv4, enter the appropriate range, and just take note of the IPv6 value because then we don't have to keep switching backwards and forwards to this document in this case it's 0, 5, paste in the IPv4 range, manual for IPv6, select the VPC range in the drop-down, down arrow twice, and then the left arrow and set to 0, 5, which is the unique value for the subnet, click add new subnet, this time it's app B, enter that name, availability zone B, manual for IPv4, you'll need to enter the IP range for app B, and again pay attention to the fact that the unique value for the subnet is 0, 6, manual for IPv6, select the VPC range in the drop-down, down arrow twice, right arrow until it says 0, 6, and then add new subnet again, we're going to do the last one, this time it's WebB, enter that name, availability zone B, and drop-down, manual for IPv4, copy and paste the IP range, and pay attention to 0, 7, which is the IPv6 unique value, enter the IPv4 subnet range in this box, manual for IPv6, select the VPC range in the drop-down, down arrow twice, and then right arrow until it says 0, 7, and now we've got all four subnets in AZB, so click on create subnet, and then we're going to do this one last time for availability zone C, so click on create subnet, select the animals for IPPC in the drop-down, and we're going to follow the same process, so for subnet 1 it will be SN_reserved-C, availability zone C, you'll need to enter the IPv4 range, pay attention to the IPv6 unique value, which is 0, 8, paste in that range in the box, manual for IPv6, select the VPC range, down arrow twice, set it to /64, and then right arrow until it says 0, 8, scroll down, add a new subnet, next is DVC, so enter that, availability zone C, and do the same thing as before, we'll need the IP range, and the IPv6 unique value, so 0, 9, enter that, manual for IPv6, select the VPC range in the drop-down, down arrow twice, and then click the right arrow until 0, 9 is selected, add a new subnet, then the application subnet, copy that, paste it in, availability zone C, get the IPv4 range and the unique value of IPv6, now note this is hexadecimal, so 0, 8 directly follows 0, 9, so pay attention to that, go back, paste in the IPv4 range, manual for IPv6, select the VPC range, down arrow twice to select /64, and then right arrow until you get 0, 8, then one last time, click on add new subnet, go back to the subnet document, Web C, availability zone C, get the IPv4 range and note the unique IPv6 value, paste that in, select the IPv6 range for the VPC, down arrow twice to select /64, and then right arrow all the way through to 0, B, at that point you can go ahead and click on create subnet, and that's created all four subnets in availability zone C, and all of the subnets now that are within the annuals for IPv6, at least those in AZA, AZB, and AZC, once again we're not going to create the ones in AZD which are reserved for future growth, now there's one final thing that we need to do to all of these subnets, so each of these subnets is allocated with an IPv6 range, however it's not set to auto-allocate IPv6 addressing to anything created within each of these subnets, now to do that go ahead and select SN-AP-A, click on actions, and then edit subnet settings, and I want you to check the box to say enable auto-assign IPv6 addresses, once you've done that scroll to the bottom and click on save, so that's one subnet that you've done that for, next I want you to do it for app B, follow the same process, actions, edit subnet settings, auto-assign IPv6, click on save, notice how we're not touching the IPv4 setting, we'll be changing that as appropriate later, select SN-AP-AP-A and see and again edit subnet settings, enable auto-assign IPv6, and click on save, then we're going to do the same for the database subnets, so dba, edit subnet settings, enable IPv6 and save, then dbb, check this box, save, then dbc, edit subnet settings, check this box, save, now we'll do the reserve, so reserve A, then reserve B, and then reserve C, and then finally we're going to do the web subnets, so we'll start with A, again make sure you're only changing the IPv6 box, say that, do the same with web B, and then once that's done we'll scroll down and do the final subnet, so web C, same process, IPv6, and save, and at this point you've gone through the very manual process of creating 12 subnets across three availability zones using the architecture that's shown on screen now, now in production usage in the real world you would automate this process, you wouldn't do this manually each and every time, but I think it's important that you understand how to do this process manually, so you can understand exactly what to select when configuring automation to achieve the same end goal, so whenever I'm using automation I always like to understand how it works manually, so that I can fully understand what it is that automation is doing, now at this point that is everything that I wanted to cover in this demo lesson, we're going to be continually evolving this design as we move through this section of the course, but at this point that is everything I wanted to do, so go ahead and complete this video and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome to this lesson where I'm going to be talking about S3 access points, which is a feature of S3, which improves the manageability of S3 buckets, especially when you have buckets which are used by many different teams or users, or when buckets store objects with a wide range of functions.

      Now we have a good amount to cover, so let's just jump in and get started.

      S3 access points simplify the process of managing access to S3 buckets and objects.

      I want you to imagine an S3 bucket with billions of objects using many different prefixes.

      Imagine this bucket is accessed by hundreds of different teams within business.

      Now by default you would have one bucket with one really complex bucket policy.

      It would be hard to manage and prone to errors.

      Access points allow you to conceptually split this.

      You can create many access points for a bucket and each of these can have different policies, so different access controls from a permissions perspective.

      But also, each access point can be limited in terms of where they can be accessed from.

      So for example, a VPC or the internet.

      Every access point has its own endpoint address and these can be given to different teams.

      So rather than using the default endpoint for S3 and accessing the bucket as a whole, users can use a specifically created access point, along with that specific endpoint address, and get access to part of that bucket or the whole bucket, but with certain restrictions.

      Now you can create access points using either the console UI, or you can use the CLI or ABI using create-access-point.

      And it's important that you remember this command.

      Please try and remember create-access-point.

      Now it's going to be easier for you to understand this architecture if we look at it visually.

      Let's explore...

      Or how everything fits together using an architecture diagram.

      And we're going to use a typical example.

      Let's say that we have an S3 bucket and this bucket stores sensitive health information for the animals for life organization.

      Now this includes health information about the animals, but also the staff working for the business, such as medical conditions and any vaccination status.

      Now we have three different sets of staff.

      We have admin staff, field workers and vet staff.

      The admin staff look after the organization's employees.

      The field workers actually visit other countries doing wildlife studies and helping animals.

      And then the vet staff look after the medical needs of any animals which the business takes care of.

      Now in this example, the business are also using a VPC, with some instances and other resources performing data analysis functions.

      If we weren't able to use access points, then we'd have to manage the bucket as a monolithic entity, managing a large and complex bucket policy to control access for identities within this account and potentially other accounts.

      And this can become unwieldy very quickly.

      One option that we have is to use access points.

      And you can sexually think of these as mini-buckets or views on the bucket.

      So we might have three access points for our three types of users and then one for the VPC.

      Now each of these access points will have a unique DNS address for accessing it and it's this DNS address that we would give to our staff.

      But more than this, each access point also has its own policy.

      And you can think about this as functionally equivalent to a bucket policy, in that it controls access to the objects in a bucket when using that access point.

      So this can control access to certain objects, prefixes or certain tags on objects.

      And so it's super powerful.

      So now we have a unique DNS name and a unique policy.

      Essentially we have mini-buckets which are independently controlled.

      And this makes it much easier to manage the different use cases for the main bucket and our staff can access it via the access points.

      From the VPC side, access points can be set to only allow a VPC origin, which means that the access point is tied to a specific VPC.

      This will need a VPC endpoint in the VPC and the two can be tied together so that the S3 endpoint in the VPC can be configured to only allow access via the S3 access point.

      Now one really crucial thing to understand permissions-wise is that any permissions defined on an access point need to be also defined on the bucket policy.

      So in this example, if the...

      That's Stafford Grant.

      And an access via an access point policy, then the same would need to also be granted via the bucket policy.

      Now you can do delegation where on the bucket policy you grant wide open access via the access point.

      So as long as the access point is used, any action on that bucket's objects is allowed.

      And then you'll be fine, more granular control over access to objects in that bucket using the access point policies.

      And that's a pretty common permissions architecture to make things simpler to manage.

      Now I've included a link attached to this lesson with more details on permissions delegation together with some example access point policies.

      You won't need this for the exam, but if you do want to do a bit of extra reading, do make sure you check out the links included with this lesson.

      Now at this point, that's everything I wanted to cover.

      You just need to have an overview of how this works.

      At this point, that's everything.

      So thanks for watching.

      Go ahead and complete this video.

      And when you're ready, I look forward to you joining me in the next.

    1. Welcome back and in this lesson I want to quickly cover S3 access logs which is a feature available within S3 which you need to be aware of.

      Now it's pretty simple to understand and so we're going to cover it pretty quickly.

      So let's jump in and get started.

      The concept of access logging is simple enough.

      We have in this example a source bucket and a target bucket.

      Now the source bucket is what we want to gain visibility up.

      So we want to understand what types of accesses are occurring to this source bucket.

      And the target bucket is where we want the logging to go.

      To use the feature we have to enable logging on the source bucket and this can be done either using the console UI or using a put bucket logging operation using the CLI or the API.

      Now logging is managed by a system known as the S3 log delivery group which reads the logging configuration which you set on the source bucket.

      It's important at this point to understand that this is a best efforts process.

      Either enabling this feature or making changes in configuration can take a few hours to take effect.

      To use this feature you also need to give the log delivery group access to the target bucket and this is done using an ACL on the target bucket.

      You add the S3 log delivery group giving it right access and this is how it can deliver the logs through the target bucket.

      Now logs are delivered as log files and each file consists of a number of log records and these and new line are limited.

      Each record consists of attributes such as date and time, the requester, the operation, status codes, error codes and much more.

      If you've seen an Apache log file then these are very similar.

      Now each attribute within a record is spaced unlimited so the records within a file are new line limited and the attributes within a record are spaced limited.

      Now a single target bucket can be used for many source buckets and you can separate these easily using prefixes in the target bucket and this is something that's configured within the logging configuration that's set on the source bucket.

      Access logging provides detailed information about the requests which are made to a source bucket and they're useful for many different applications, most commonly security functions and any access audits and it can also help you to understand the access patterns of your customer base and understand any charges on your S3 bill.

      If you do use this feature then you need to personally manage the lifecycle or deletion of any of the log files.

      This is not built into the product so you need to manage either the movement of these log files to different storage classes or deleting them after a certain amount of time.

      Now that's it for the architecture.

      If you're doing a course where you do need practical experience then I'll be following this up with a demo.

      If not then this theory is all that you'll need.

      At this point, thanks for watching.

      Go ahead and complete this video and when you're ready I look forward to you joining me in the next.

    1. Welcome to this lesson where I want to step through the event notification feature of S3.

      This is a feature which allows you to create event notification configurations on a bucket.

      So let's jump in and get started exploring the architecture of this feature and exactly how we can use it.

      The feature is pretty simple to understand.

      When enabled, a notification is generated when a certain thing occurs within a bucket.

      And these can be delivered to different destinations, including SNS topics, SQSQs or LAN functions.

      And this means that you can have event-driven processes which occur as a result of things happening within S3.

      Now, there is different types of events supported.

      For example, you can generate event notifications when objects are created, which means put, post, copy and when long multi-part upload operations complete.

      Maybe you want to do something like take an image and add a watermark or do something crazy like generating a retro pixel art version of any images uploaded to a bucket.

      You can create an event notification configuration, set it so that it triggers whenever an object is created, send this to some destination to process the image, and then you have an event-driven automated workflow.

      You can also set event notifications to trigger on object deletion so you can match any type of deletion using the Star Wild card.

      You can match delete operations or when delete markers are created.

      And you might use this if you want an automated security system to react if any objects are removed from any of your S3 buckets.

      You can also have it trigger for object restores, so if you have objects in S3, Glacier or Glacierd Part 5 and you perform a restore operation, you can be notified when it starts and completes.

      And this can be useful if you want to notify customers or staff when a restore begins and ends.

      Then finally, you can get notifications relating to replication.

      So if operations miss the 15-minute threshold, if they're replicated after the threshold, when an object is no longer tracked, and even if an object fails replication, all of those things can generate an event notification using this product.

      Now, visually, it looks like...

      Like this.

      We start with an S3 bucket and define... ...an event notification configuration.

      The configuration matches events which we want to be notified of.

      Overtide these events occur which, as I just talked about, include create events, delete events, restore events and replicate events.

      And when these things occur, events are generated and sent to the destinations which you configure.

      And at the time of creating this lesson, that's Lambda, SQSQs and SNS topics.

      These events interact with those services.

      Events are generated by the S3 service known as the S3 principle.

      And so we need to also add resource policies onto each of those destination services allowing the S3 service to interact with them.

      This could be in the form of an SQSQ policy or a Lambda resource policy.

      And at the time of creating this lesson, the only way to modify Lambda resource policies is to use the CLI or the API.

      Although this might change over time.

      The events themselves are actually JSON objects.

      And this is a cut-down version just to give you an idea of what they look like.

      Now, there are bits missing, I know, but you'll get the general idea.

      If you use Lambda, for example, this is received within the event structure and you'll need to pass that, extract the information that you need and then act on it.

      Now, S3 event notifications are actually a relatively old feature and support only a limited set of things occurring on objects in S3 buckets and can only interact with a limited number of AWS services.

      You can also use EventBridge, which supports more types of events and can integrate with a wider range of AWS services.

      As always with AWS, you have a few different ways of doing the same thing.

      Now, as a default, I would tend to lean towards using EventBridge unless you have a specific reason not to do so.

      But I did want to make sure that you understand the event notification feature of S3.

      Now, at this point, that's everything I wanted to cover.

      So go ahead and complete this video.

      And when you're ready, I look forward to you joining me in the next.

    1. Welcome back in this lesson, I want to cover a really important feature of S3 in Glacier that you need to be aware of for the exam.

      Now you don't need to understand the implementation just the architecture.

      S3 Select and Glacier Select are ways that you can retrieve parts of objects rather than the entire object.

      Now I expect that this will feature only in the minor way in the exam and it will be focused on architecture and features but I do want you to be aware of exactly how this feature works.

      So let's jump in and step through this architecture and what benefits it provides.

      Now you know by now that both S3 and Glacier are super scalable services.

      You can use them both to store huge quantities of data.

      S3 for example can store objects up to five terabytes in size and can store an infinite number of those objects.

      Now often when you're interacting with objects inside S3 you intentionally want to interact with that full object so you might want to retrieve that full five terabyte object.

      What's critical to understand as a solutions architect is that logically if you retrieve a five terabyte object then it takes time and it consumes that full five terabytes of transfer.

      So if you're downloading a five terabyte object from S3 into your application then you consume five terabytes of data you're accessing that full object and it takes time.

      Now you can filter this on the client side but this occurs after the figurative damage is done.

      You've already consumed that capacity, you've already downloaded that data, filtering it at the client side just means throwing away the data that you don't need.

      S3 and Glacia provide services which allow you to access partial objects so that's what's provided by the S3 select and Glacia select services and the way that you do this is that both services allow you to create a SQL like statement so cut down SQL statement.

      So you create this, you supply it to that service and then the service uses this SQL like statement to select part of that object and this part and only this part is sent to the client in a pre-filtered way so you only consume the pre-filtered part of that object, the part that you select so it's faster and it's cheaper.

      Now both S3 select and Glacia select allow you to operate on a number of file formats with this level of functionality so examples of this include comma separated values JSON can even use visa to compression comma separated values and for JSON so it's a really flexible service.

      Now visually this is how it looks.

      Now at the top we have the architecture without using S3 or Glacia select and at the bottom we have the architecture when we utilize these services so in both cases we have an application which stores its data on S3.

      So when an application interacts with S3 to retrieve any data the entire object or stream of objects are delivered to the application so the application receives everything.

      It can either accept it as a whole or it can perform its own filtering but the critical thing to understand is that any filtering performed by the application is performed inside the application.

      It doesn't impact the cost or performance.

      The only data which is filtered out is simply discarded but it's still billed for and it still takes time.

      Now contrast this to using S3 select with the same architecture so we still have the same application that interacts with the same S3 bucket.

      It places the filter point inside the S3 service itself and this allows us to use a SQL-like expression and provide this to the S3 select service.

      The S3 select service can then use the SQL-like expression and it can apply this to the raw data in S3 so in effect it's taking the raw data filtering it down but this is occurring inside the S3 service so the data that the application receives is the pre-filtered data and this means that we can achieve faster speeds and a significant reduction in cost.

      Once the filtering occurs before it's transferred to our application it means that we get substantial benefits both in speed and in cost and this is because the S3 service doesn't have to load all of the data and deliver it to the application.

      We're applying this filter at the source, the source of the data which is the S3 service itself.

      Now this is a feature which our applications will need to explicitly use but as a solutions architect it's a powerful feature that you need to understand to improve the levels of performance in any systems that you design.

      Now at this point that's everything I wanted to cover on to keep it brief because it's a product that I only expect to feature in a very very minor way in the exam and I do want you to be aware of its existence.

      So thanks for watching, go ahead, complete this video and when you're ready I look forward to you joining me in the next lesson.

    1. Welcome back.

      And in this lesson, I want to talk about S3 Object Lock.

      Now, this is something which is really important to understand for the SysOps certification, but equally, if you're developing applications running in AWS or architecting solutions for AWS, you also need to have an awareness.

      Now, we've got a fair amount to cover, so let's jump in and get started.

      S3 Object Lock is actually a group of related features which I'll talk about in this lesson, but it's something that you enable on new S3 buckets.

      If you want to turn on Object Lock for existing buckets, then you need to contact AWS support.

      And this is probably going to change over time, but at the time of creating this lesson, this is the current situation.

      Now, when you create a bucket with Object Lock enabled, versioning is also enabled on that bucket.

      Once you create a bucket with Object Lock and enable it, you cannot disable Object Lock or suspend versioning on that bucket.

      Object Lock implements a right once, read many architecture, and this is known as Worm.

      It means that you can set it so that Object Versions once created can't be overwritten or deleted.

      And just to reiterate, since this is a pretty important point, the feature requires versioning to be enabled on a bucket.

      And because of this, it's actually individual Object Versions which are locked.

      Now, when we talk about Object Lock, there are actually two ways it manages Object Retention.

      Retention periods and legal holds.

      An Object Version can have both of these, one or the other or none of them.

      And it's really important for the exam and for the real world to think of these as two different things.

      So, S3 Object Lock Retention and S3 Object Lock Legal Hold.

      Now, just like with bucket default encryption settings, these can be defined on individual Object Versions, or you can define bucket defaults for all of the Object Lock features.

      Now, this is just the feature at the high level.

      Next, I want to quickly step through the key points of the different retention methods.

      So, Retention Period and Legal Hold.

      And we're going to start with Retention Period.

      With the Retention Period style of object locking, when you create the Object Lock, you specify a Retention Period in days and/or years.

      One year means the Retention Period will end one year from when it's applied.

      Now, there are two modes of Retention Period Lock which you can apply.

      And it's really, really important that you understand how these work and the differences between the two.

      One, because it matters for the exam, and two, because if you get it wrong, it will cause a world of pain.

      The first mode is Compliance Mode.

      If you set a Retention Period on an object using Compliance Mode, it means that an Object Version cannot be deleted or overwritten for the duration of the Retention Period.

      But, it also means that the Retention Period itself cannot be reduced and the Retention Mode cannot be adjusted during the Retention Period.

      So, no changes at all to the Object Version or Retention Period settings.

      And this even includes the account root user.

      So, no identity in the account can make any changes to Object Versions, delete Object Versions, or change the Retention Settings until the Retention Period expires.

      So, this is serious business.

      This is the most strict form of Object Lock.

      Don't set this unless you really want that Object Version to stay around in its current form until the Retention Period expires.

      Now, you've used this mode as the name suggests for compliance reasons.

      An example of this might be medical or financial data.

      If you have compliance laws stating that you have to keep data, for example, for three years, with no exceptions, then this is the mode that you set.

      Now, a less strict version of this is Governance Mode.

      With this mode, you still set a Retention Period, and while active, the Object Version cannot be deleted or changed in any way.

      But you can grant special permissions to allow these to be changed.

      So, if you want a specific group of identities to be able to change settings and Object Versions, then you can provide them with the permission S3 colon Bypass Governance Retention.

      And as long as they have that permission and they provide a header along with their request, which is X-AMZ-BYPASS-GOVERNANCE-RETENTION, then they can override the Governance Mode of Retention.

      Now, an important point to understand is this last header, so X-AMZ-HIVEN, and then all the rest.

      This is actually the default for the console UI.

      And so, using the console UI, you have the S3 colon Bypass Governance Retention, you will be able to make changes to Governance Mode Retention Locks.

      So, Governance Mode is useful for a few things.

      One, if you want to prevent accidental deletion.

      Two, if you have process reasons or governance reasons to keep Object Versions.

      Or lastly, you might use it as a test of settings before picking the Compliance Mode.

      So, that's Governance Mode.

      These are both modes that can be used when using the Retention Period feature of S3 Object Locking.

      So, please make sure you understand how they both work and the differences between the two before we finish with this lesson.

      It's really, really critical that you understand.

      Now, the last overview that I want to give is S3 Object Lock Legal Hold.

      With this type, you don't actually set a Retention Period at all.

      Instead, for an Object Version, you set Legal Hold to be on or off.

      To repeat, there's no concept of retention.

      This is a binary.

      It's either on or off.

      While Legal Hold is enabled on an Object Version, you can't delete or overwrite that specific Object Version.

      An extra permission is required, which is S3 colon Put Object Legal Hold.

      And this is required if you want to add or remove the Legal Hold feature.

      And this type of Object Locking can be used to prevent accidental deletions of Object Versions, or for actual legal situations when you need to flag specific Object Versions as critical for a given case or a project.

      Now, this point, let's take a moment to summarise and look visually at how all these different features work.

      Let's start with Legal Hold.

      We start with a normal Object and we upload it to a bucket with a setting Legal Hold Status to On.

      And this means that the Object Version is locked until the Legal Hold is removed.

      In this state, the Object Version can't be deleted or changed, but you can set the Legal Hold Status to Off, at which point normal Commissions apply and the Object Version can be deleted or replaced as required.

      It's a binary.

      It's either on or off, and that isn't the concept of Retention Period.

      Next, we have the S3 Object Locks that use the Retention Period architecture.

      First, we have Governance, so we put an Object into a bucket with a Lock Configuration of Governance and specify a Retention Period.

      This creates a locked Object Version for a given number of days or years, and while it's in this state, it cannot be deleted or updated.

      With Governance Mode, this can be bypassed if you have the permissions and specify the correct header.

      And once again, this header is the default in the console, so you can adjust or remove the Object Lock or delete or replace the Object Version.

      So the important thing to realise here is while an Object is locked for a given Retention Period using the Governance Mode, you can't make any changes to Object Versions or delete them, but you can be provided with the S3 colon bypass and Governance Retention Permission, and as long as you have that and specify the X-AMZ-VIPAS-GOVERNANCE-RETENTION-TRUEHEADER, then you can override the Governance Mode Object Lock during the Retention Period.

      Then lastly, we have Compliance, which is the same architecture.

      We upload an Object.

      We specify Compliance Mode together with the Retention Period, and this creates an Object Version, which is locked for a certain period in days and years.

      The difference though is that this can't be changed.

      An Object Version can't be deleted or updated.

      The Retention Period cannot be shortened.

      The Compliance Model can't be changed to something else even by the account root user.

      This is permanent.

      Only once the Retention Period expires can the Object Version or the Retention Settings be updated.

      And for all of these, they can be set on Object Versions or what defaults can be fine.

      And that's the architecture of S3 Object Lock.

      It's critical that you understand this.

      If it takes a few watches at this lesson, then that's OK.

      Make sure you understand it in detail, including how each type differs from the others.

      And remember, they can be used in conjunction with each other, so the effects can overlap.

      You might use Legal Hold together with either Governance or Compliance, and if you do, then the effects of this could overlap, so you need to understand all of this in detail.

      But at this point, that's everything I wanted to cover in this lesson, so go ahead and complete the video.

      I mean, you ready?

      I look forward to you joining me in the next.

    1. Welcome to this mini project where you're going to get the experience of creating S3 multi-region access points.

      Now multi-region access points give you the ability to create a single S3 global endpoint and point this at multiple S3 buckets.

      It's an effective way to use a single endpoint throughout requests to the closest S3 service.

      Now in order to do this mini project you need to be logged in to an AWS account with admin permissions.

      If you're using one of my courses then you should use the IAM admin user of the general AWS account which is the management account of the organization.

      If you're not using my courses make sure you're using an identity with admin permissions.

      You'll also need to select two different AWS regions before this mini project we're going to create S3 buckets in two regions.

      I'm going to use AP, South East 2 or the Sydney region and CA Central 1 or the Canada region.

      Now the first thing to do is to move to the S3 console so type S3 in the search box at the top and then open that in a new tab.

      Once you're there we're going to create two buckets so first go ahead and click on create bucket.

      Now we'll keep the bucket naming consistent so we'll use multi-heiven region -demo -heiven and then the region that you're in.

      So in my case Sydney and then at the end I want you to append on a random number.

      In my case 1-3-3-7.

      Remember S3 bucket names need to be globally unique and this will ensure both of our buckets are.

      Once you put the name make sure you set the region correctly everything else can be left as default apart from we need to enable bucket versioning.

      So set this box under versioning to enable.

      Now scroll to the bottom and click on create bucket.

      Then we need to follow that same process again for the second bucket.

      So click on create bucket.

      Use the same bucket naming so multi-heiven region -demo -heiven and then the region name in this case Canada.

      And make sure you append your random number and set the region.

      Then scroll down, enable bucket versioning again and create the bucket.

      Now once you've got these two buckets we're going to create the multi-region access point.

      So click on multi-region access point on the menu on the left and click create multi-region access point.

      For the name you can pick whatever you want it doesn't need to be globally unique only unique within an AWS account.

      I'm going to pick really really critical cat data and then scroll down and add the buckets that you've just created.

      These can't be added or edited after creation so we need to do it now.

      Now we're going to add buckets.

      Select the two buckets and then click on add buckets to confirm.

      Once you've done that scroll down to the bottom and click create multi-region access point.

      Now this process can take worst case up to 24 hours to complete but typically it creates much faster, generally around 10 to 30 minutes.

      Now we do need this to be created before we continue.

      So go ahead and pause the video, wait for the status on this to change to ready and then you're good to continue.

      Okay so now that we've got this multi-region access point configured and it's ready to go.

      Now that we've got this multi-region access point configured we need to configure replication between the two buckets because anyone using this multi-region access point will be directed to the closest S3 bucket and we need to make sure that the data in both matches.

      So to do that go ahead and click on the multi-region access point name and go inside there and you'll see that the access point has an Amazon resource name as well as an alias.

      Now you should probably note down the Amazon resource name because we might need it later on.

      Once you've done that click on the replication and failover tab and you'll be able to see a graphical representation of any replication or failover configuration.

      If we click on the replication tab you'll see there's no replication configured.

      If we click on the failover tab you can see that we've got these two S3 buckets in different AWS regions configured as an active active failover configuration which means any requests made to this multi-region access point will be delivered to either of these S3 buckets as long as they're available.

      Now we can click on one and click on edit routing status and configure it as passive which means it will only be used if no active buckets exist.

      But in our case we want it to be active active so we'll leave both of these set to active.

      Now we want to configure replication between the buckets so we're going to scroll down to replication rules and click create replication rule.

      Now there are two templates available to start with, replicate objects amongst all specified buckets and replicate objects from one or more source buckets to one or more destination buckets.

      Now which of these you pick depends on the architecture that you're using but because we have an active active configuration we want all the buckets to be the same.

      So we're going to pick the replicate objects among all specified bucket template so this is replicating between every bucket and every other bucket.

      Essentially it creates a set of buckets which contain exactly the same data all fronted by a multi-region access point.

      So go ahead and make sure this template is selected and then click to select both of the buckets that you created.

      In my case Sydney and Canada.

      Once we've done that scroll down you can set whether you want the status to be enabled or disabled when created we're going to choose enable and you get to adjust the scope so you can either have it configured so that you can replicate objects using one or more filters or you can apply to all objects in the bucket.

      Now we want to make sure the entire bucket is replicated so we're going to use apply to all objects in the bucket.

      Now you're informed that an Ion role or roles will be generated based on your configuration and this will provide S3 with the permissions that it needs to replicate objects between the buckets.

      Now this is informational we don't need to do anything so let's move on.

      Now you're also told what encryption settings are used as well as the destination storage class so because of the template that we picked above we don't get to change the destination storage class and that's okay.

      If we scroll down to the bottom we have additional replication options, we have replication time control which applies in SLA to the replication process, we have replication metrics and notifications to provide additional rich information and we can choose whether to replicate the lead markers and whether to replicate modifications.

      Now for this mini project we're not going to use replication time control we don't need that level of SLA.

      We are going to make sure that replication metrics and notifications is selected.

      We don't want to replicate the lead markers and we do want to make sure that replica modifications sync is checked.

      So we only want replication metrics and notifications and replica modifications sync.

      So make sure that both of those are checked and then click on create replication roles.

      Now at this point all the buckets within this multi-region access point are now replicating with each other.

      In our case it's only the two, in my case it's Canada and Sydney.

      So go ahead and click on close and we can see how this graphical representation has changed showing us that we now have two-way replication in my case between Sydney and Canada.

      Now at this point we need to test out the multi-region access point and rather than having you configure your local command line interface we're going to do that with Cloud Shell.

      Now what I want you to do is to go ahead and move to a different AWS region so not the same AWS regions that either of your buckets are created in.

      What I do want you to do though is make a region close to one of your buckets.

      Now I'm going to start off with Sydney and in order to test this I'm going to switch across to the Tokyo region which is relatively close to Sydney, at least from a global perspective.

      So I'm going to click on the region drop down at the top and change it from Sydney to Tokyo.

      And what's on there I'm going to click on this icon which starts the Cloud Shell.

      If this is the first time you're using it in this region you'll probably get the welcome to AWS Cloud Shell notification.

      Just either click on close or check this box and then click on close if you don't want to see this notification again.

      Now all these commands that we're going to be running are in the instructions which are attached to this video.

      The first thing that we're going to do is to create a test file that we're going to upload to S3.

      We're going to do that using the DD command.

      So we're going to have an input of /dev/urandom which just gives us a stream of random data.

      And then for the output using the OF option we're going to create a file called test1.file.

      This is going to have a block size of 1 meg and a count of 10 which means that it's going to create a 10 meg file called test1.file.

      So run that command.

      Now once you've done that just go back to the tab that you've got open to S3.

      Scroll to the top, click on multi-region access points, check the access point that you've created and then just click on copy ARN to copy the ARN for this access point into your clipboard and then go back to Cloud Shell.

      Next I'm going to do an LS making sure I just have a file created within Cloud Shell, I do.

      And now I'm going to run this command so aws space S3, space CP for copy, space test1.file.

      So this is the local file we created within Cloud Shell and then space and then S3 colon, double forward slash and then the ARN of the multi-region access point.

      Now this command is going to copy the file that we created to this multi-region access point and this multi-region access point is going to direct us towards the closest S3 location that it serves which should be the bucket within the Sydney region.

      So go ahead and run that command.

      It's going to take a few moments to upload but when it does switch back to S3, go to buckets.

      In my case I'm going to go to the Sydney bucket and I should see the file created in this bucket.

      I do, that's good, so I'm going to go back to buckets and go to Canada and I don't yet see the object created in the Canada bucket and that's because replication can take a few minutes to replicate from the bucket where the object was stored through to the destination bucket.

      If we just give this a few moments and keep hitting refresh, after a few moments we should see the same S1.file which has been replicated from the Sydney region through to the Canada region.

      Now S3 replication isn't guaranteed to complete in a set time, especially if you haven't configured the replication time control option.

      So it's fairly normal to see a small delay between when the object gets written to one bucket and when it's replicated to another.

      Now we're going to try this with a different region, so go back to Cloud Shell and then click on the region drop down and we're going to pick a region which is close to our other bucket but not in the same region.

      So the other bucket is created in the Canada region, so I'm going to pick a close region.

      In this case I'm going to pick US East 2 which is in Ohio.

      Once I've done that I'm going to go back to Cloud Shell.

      Once I've done that I should be in Cloud Shell in a different AWS region.

      So I'll need to recreate the test file.

      In this case I'm going to call it test2.file and I'm going to use all the same options.

      And again this command is contained in the instructions attached to this video.

      So run that command and it will take a few moments to complete and then we're going to follow the same process.

      We're going to upload this file to our S3 buckets using the multi-region access point.

      So again just make sure you've got the ARN for the access point in your clipboard and then in the Cloud Shell type AWS space S3, space CP, space test2.file, space S3, colon, forward slash forward slash and then the ARN of the multi-region access point.

      Again we're going to press enter, wait for this to upload and then check the buckets.

      So run that command and then go back to S3, go to buckets.

      I'm going to go to the bucket in Canada first and I'm going to hit refresh and we should see test2.file in this bucket which is good.

      Then go to buckets and go to Sydney and we probably won't see that file just yet because it will take a few moments to replicate.

      So keep hitting refresh and eventually you should see test2.file arrives in the S3 bucket.

      Now this time we're going to run the same test but we're going to pick a region that is relatively in the middle between these two regions where our S3 buckets are created.

      In my case I'm going to change the region to APSALV1 which is the Mumbai region.

      And I'm going to follow exactly the same process and move to Cloud Shell.

      I'm going to generate a new file so in this case it's test3.file.

      Now before we upload this we're going to go to the S3 console, go to buckets and just make sure that we have a tab open for both the Canada and the Sydney bucket.

      This test is going to be uploaded from a region that's in the middle of these two buckets and so we need to be able to quickly check which bucket receives the upload directly and which bucket receives the replicated copy.

      So once you've got a tab open for both those buckets, go back to Cloud Shell and run this command.

      So aws space S3 space CP, space test3.file, space S3//// and then the name of the S3 multi-region access point.

      Once you've done that go ahead and run that command, it will take a few moments to upload and then move straight to the tabs that you've got open to the S3 buckets and refresh both of them.

      If I look at Sydney it looks as though that receives the file straight away.

      So this multi-region access point is directed us to Sydney.

      If I move to Canada and hit refresh we can see that the file's not yet arrived so this is receiving the replicated copy.

      And it does that after a few minutes we can see test3.file arrives in this bucket.

      Now this one we're going to run a test to show the type of problem that can occur if you're using this type of global replicated configuration using multi-region access points.

      What I want you to do is to open up two different Cloud Shells.

      I want one Cloud Shell open in the Canada region so the region may have one of your buckets and I want the other Cloud Shell open in the AP Southeast two region, so the Sydney region.

      So we've got one Cloud Shell open in the same region as each of our buckets.

      Now once we've done that in one region, in my case I'm going to use the Canada region, I'm going to generate a file called test4.file.

      The command is going to be the same as we've used in previous steps.

      Then I'm going to copy the ARN of the multi-region access point into my clipboard and I'm going to type this command again in the Canada Cloud Shell.

      So this command is going to copy this file to the multi-region access point.

      Now because I'm doing this in the Canada region, it's almost guaranteed that the multi-region access point is going to direct us to the bucket which is also in the Canada region.

      And so the bucket in this region is going to get the direct copy of the object and the bucket in the Sydney region is going to receive the replicated copy.

      So I'm going to fully type out this command and before I run it I'm going to move it to the Sydney region and type out a slightly different command.

      This command is also contained in the instructions attached to this lesson.

      This command is AWS space S3 space CP space and then S3 colon forward slash forward slash and then the name of the multi-region access point and then forward slash test4.file and then space and then a period.

      Now this command when we run it which we're not going to do yet is going to copy the test4.file object from the multi-region access point into our Cloud Shell.

      Remember we haven't created this object yet.

      Now because this Cloud Shell is in the Sydney region, the multi-region access point is almost certainly going to redirect us to the bucket in the Sydney region.

      So let's move back to the Canada Cloud Shell, run this command which is probably going to copy this object into the Canada bucket, then move back to the Cloud Shell in the Sydney region and then run this command to copy the object from the multi-region access point into our Cloud Shell and we receive this error.

      Now we're getting this error because the object test4.file doesn't exist within the bucket that the multi-region access point is directing us to.

      So just to reiterate what we've done, we've created this object in the Canada region using the multi-region access point which is going to have created the object in the closest S3 resource which is the bucket also in the Canada region.

      So the Canada region bucket has the direct copy of the object.

      This will then take a few minutes to replicate to the Sydney bucket because we're using this reverse copy command in the Sydney region.

      It's attempting to copy the test4.file object from the multi-region access point to our Cloud Shell but because this replication won't have occurred yet and because this is going to direct us at the bucket in the Sydney region, we get this object does not exist error.

      And this is one of the issues you can experience when you're using multi-region access points in that there's a consistency lack.

      You have to wait for the replication to occur before you can retrieve replicated objects from different buckets using the multi-region access point.

      Now this process can be improved by using the RTC option when setting up replication but this does come with additional costs.

      So this is something to keep in mind when you're using this type of architecture.

      And if we keep running this command we'll see that it keeps failing over and over again until the replication process finishes and then we can copy down the object.

      Now that is everything which I wanted to cover in this brief mini project in the multi-region access points.

      At this point all that remains is for us to clean up the account and return it to the same state as it was at the start of this mini project.

      So to do that we need to move back to the S3 console, go to multi-region access points, select the access point you created for this mini project and click delete.

      And then you need to copy and paste the name and click delete to confirm.

      Then we need to go ahead and move to buckets.

      Select each of the buckets in turn, first click on empty and you need to copy and paste or type permanently delete and then click to confirm that empty process.

      Once that's finished click on exit, select the same bucket again, this time click delete, copy and paste or type the bucket name and then click to confirm.

      And then do the same process with the other buckets so first empty it, confirm that and then delete it and confirm that.

      And then at that point all the billable elements created as part of this mini project have been deleted and we go to finish off this mini project.

      So that's everything I wanted to cover, I hope it's been useful and I hope it's given you some practical experience of how to use multi-region access points together with S3 replication.

      At this point that's everything to go ahead and complete this video and I hope you'll join me soon for another exciting mini project.

    1. Welcome back.

      In this brief demo lesson, you're going to get some experience working with S3 pre-signed URLs.

      Now, as you learned in the theory lesson, a pre-signed URL is a type of URL which can be used to grant access to certain objects within an S3 bucket where the credentials for accessing that object are encoded on the URL.

      I want to explain exactly what that means and how you can use it in the real world.

      Now, to do that, we're not going to need to create any infrastructure using CloudFormation.

      Instead, we're going to do it manually.

      So first, I want you to make sure that you're logged in to the general AWS account as the I am admin user.

      And as always, please make sure that you have the Northern Virginia region selected.

      Assuming that's all good, go ahead and type S3 in the search box at the top and open that in a new tab.

      We're going to create an S3 bucket within the general AWS account.

      So go ahead and click on create bucket to be in that process.

      Now, I want you to call the bucket animals for life media.

      And because of the unique naming requirements of S3 buckets, you'll need to add some randomness onto the end of this name.

      We'll need to select us - east - one for the region.

      We can scroll past the bucket settings for block public access.

      We don't need to enable bucket versioning.

      We won't be using any form of encryption.

      Just go ahead and click on create bucket.

      Now I want you to go inside the bucket that you've created and click on the upload button to upload an object.

      Now, at this point, we need to upload an image file to this bucket.

      Any image file will do.

      But I've included a sample one attached to this lesson if you don't have one available.

      So click the link for the image download.

      That will download an image called all5.jpeg.

      Once that's downloaded to your local machine, click on add files, select the file, open it, and then go ahead and upload the file to the S3 bucket.

      Once that's finished, you can go ahead and click on close.

      And you'll see now that we've got our S3 bucket and one object uploaded to that bucket called all5.jpeg.

      Next, go ahead and click on that object.

      Now, I want to demonstrate how you can interact with this object in a number of different ways.

      And the detail really matters here.

      So we really need to be sure of exactly what differences there are between these different methods.

      The first thing I want you to do is towards the top right of the screen, click on the open button.

      You might get a pop-up notification if you do just allow pop-ups.

      And what that's going to do is open a jpeg object in a new tab.

      Now, I want to point out a number of really important points about how this object has been opened.

      If you take a moment to review the URL that's been used to open this object, you'll note that a number of pieces of information have been specified on the URL, including AMZ-SecurityToken.

      So essentially, a form of authentication has been provided on the URL which allows you to access this object.

      Now, I want you to contrast this by what happens if we go back to the S3 bucket and just copy down the object URL into our clipboard and note how this does not contain any additional authentication information.

      It's just the raw URL for this object.

      So copy that into your clipboard and then open a new tab and paste that in.

      What you're going to see is an access denied message.

      And this makes sense because we're now attempting to access this object as an unauthenticated identity, just like any internet user, anyone browsing to this bucket would be doing.

      We're not providing any authentication.

      And so the only way that we can access the object in this way by not providing any authentication is if we made the bucket public and the bucket currently isn't public, which is why we're getting this access denied message.

      Just to reiterate, we can access it by using the previous method because by opening it from within the console, the console is intelligent enough to add authentication information onto the URL which allows us to access this object.

      So those are the differences between these two methods.

      One is providing authentication and the other isn't.

      So right now the only entity that is able to access any of the objects within this bucket is this AWS account and specifically the I am admin user.

      The object has no public access.

      So now what we're going to do is we're going to work with the scenario that you want to grant access to this object to somebody else for a limited amount of time.

      So you don't want to provide the URL that includes authentication information, you want to provide a URL which allows access to that object for a limited amount of time.

      That's important.

      So we're going to generate a time limited pre-sign URL.

      So go ahead and click on the Cloud Shell icon and this is going to open a Cloud Shell using the identity that you're currently logged in at.

      So it's going to open a shell much like the one you'll see when you're connected to an EC2 instance but the credentials that you have in the shell are going to be the credentials of the identity that you're currently logged into AWS using in our case the I am admin user with a general AWS account.

      You'll see a message saying preparing your terminal and then you'll be logged in to what looks like an EC2 instance prompt.

      You can use the AWS CLI tool so I'm able to run an AWS space S3 space LS and this shell will interact with your current AWS account using your current credentials.

      So in my case I'm able to see the Animals for Life media bucket which is in my general AWS account because I'm currently logged into this Cloud Shell using the credentials of my I am admin user.

      Now to generate a pre-sign URL we have to use this command.

      So use AWS to use the command line tools and then a space S3 because we're using the S3 service and then a space and then the word pre-sign because we want to generate a pre-signed URL and then a space and then we need the S3 URI to this object.

      Now we can get that from the S3 console.

      For every object you can see this unique URI so go ahead and copy this into your click mode go back to the Cloud Shell paste it in and then we'll need space and then we'll use double hyphen expires hyphen in and then a space and then we need to provide the number of seconds that this pre-signed URL will be valid for.

      In this case we're going to use 180 which is a total of three minutes so this is in seconds so three minutes is 180 seconds.

      So go ahead and press enter and this will generate you a unique pre-signed URL.

      So go ahead and copy this into your click board and it is a really long URL so make sure that you get everything including HTTPS all the way to the end of this URL.

      Copy that into your click board and then I want you to open a new tab and load that URL and there you go you can see this object loads up using this pre-signed URL.

      Now this URL is valid only for 180 seconds.

      To demonstrate that I'm going to skip ahead 180 seconds and demonstrate exactly what happens when this URL expires.

      After 180 seconds when a next refresh we see this access denied page with a message request has expired.

      So you can see how pre-signed URLs are a really effective way of granting access to objects within an S3 bucket for a limited amount of time.

      Now there are a number of really interesting aspects to pre-signed URLs that you really need to understand as an architect, a developer or an engineer and I want to go through these really interesting aspects of pre-signed URLs before we finish up with this demo lesson.

      Now first just to make everything easier to see I'm going to close down any of these tabs that we got open to this all five object.

      I'm going to go back to the cloud shell and I'm going to generate a new pre-signed URL but this time I'm going to use a much larger expires in time.

      So I'm going to press the up arrow to return to the previous command.

      I'm going to delete this 180 second expiring time and instead I'm going to use 604,800 which is a pretty high number but this is something that we can pick so that we won't have any unintentional expires at the URL as part of this demo lesson.

      So pick something crazily large just to make sure that it doesn't expire until we're ready.

      So we're generating another URL and so this is an additional unique pre-signed URL.

      So I'm going to select all of this URL and you need to do the same go ahead and copy that into your clipboard and open it in a new tab.

      Now we can see that that pre-signed URL has opened.

      Keep in mind that we've generated this using the identity that we're currently logged into AWS using.

      So next what I'm going to do is move back to the AWS console and I'm going to click on the services drop down and move across to IAM.

      So I'm going to open IAM up in a new tab.

      I'm going to select the users option, select the IAM admin user, I'm going to add an inline policy to this user, select JSON.

      Now attached to this lesson is another link for a file called deny s3.json.

      Go ahead and click that link and then you'll need to get the contents of the file.

      So copy all the contents of the file into your clipboard and then select all of this JSON and paste in the contents of that file and this is an explicit deny policy which denies our user so IAM admin any access to s3.

      So this essentially prevents our IAM admin user any level of access to s3.

      Go ahead and click on review policy for name, call it deny s3 and click on create policy and this will attach this as an inline policy to our IAM admin user.

      Now I'm going to clear the screen within this cloud shell to make it easier to see and then I'm going to run an AWS space s3 space ls and press enter.

      Now we get an access deny when accessing the s3 service because we just added an explicit deny onto the IAM admin user and remember the rule that applies to permissions deny allow deny and explicit deny always overrules everything else and so even though the IAM admin user has administrative permissions by applying this deny s3 policy which has an s3 explicit deny this wins so currently we have no access to s3.

      Now let's return to the tab where we have this pre-signed url open.

      So remember this pre-signed url was generated at the time when we had access to s3 so now let's refresh this page and now we get an access denied message and this is one of those interesting aspects of pre-signed urls.

      When you generate a pre-signed url using an identity such as an IAM user that url has your access permissions so you generate a pre-signed url and anyone using that pre-signed url for the duration that it's active will be interacting with that one object on s3 as though they were using your identity.

      Now if you adjust the permissions on your identity as we've just done by denying access to s3 it means that that pre-signed url will also have that deny s3 permissions and so this pre-signed url now no longer has any access to anything in s3 including the object that it was configured to provide access to so now let's go back to our cloud shell and what we're going to do now remember we are still denied access to s3.

      Let's press the app arrow and move back to the command which we used to generate this pre-signed url.

      Now let's regenerate another pre-signed url.

      Now note how even though we are denied access to any part of s3 we can generate a pre-signed url which points at this one specific object all five dot jpeg so we're not prevented from generating a pre-signed url for something that we have no access to.

      Now if we copy this into our clipboard move to the tab that we already have open and just replace this url with the pre-signed url that you just generated.

      Note that we're still denied we're still denied because although we could generate this pre-signed url we've generated it using an identity which has no access to s3 so the next interesting fact to pre-signed urls that I want to demonstrate if we go back to the iam console and now we remove this deny s3 policy from our iam admin user now our iam admin user once again has access to s3 so if we go back to the cloud shell we can run natal us space s3 space ls and press enter and now we can access s3 again from this cloud shell remember the cloud shell is using commissions based on our iam admin user now let's go back to the pre-signed url and now if we refresh this now we can access this object again so I'm doing this to illustrate that when you generate a pre-signed url that pre-signed url is linked to the identity that generates it whatever the permissions are of that identity when the pre-signed url is used that is the permissions that that pre-signed url has so for the duration it always has the same permissions that the identity which generated it has at that very moment so that's something that you need to keep in mind now more interestingly is that you can actually generate a pre-signed url for a non-existent object there's nothing preventing that from occurring so if we go back to the cloud shell we press the up arrow and a number of times to bring up this pre-signed url command and this time we try to generate a pre-signed url for an object which doesn't exist so if I change this all 5 to all 1 3 3 7 and press enter it will generate a pre-signed url that pre-signed url will be valid to access an object called all 1 3 3 7 .jpeg inside this bucket but because there's no such object in that bucket if I try to use it I won't be able to do so so if I open that new invalid pre-signed url I'll get the message that the specified key does not exist but we can generate pre-signed urls for non-existent objects now one more interesting thing about pre-signed urls which I'm not going to demonstrate is if you generate a pre-signed url using temporary credentials you get by issuing a roll so for example if we logged into an ec2 instance which had an instance roll on that instance and then we generated a pre-signed url even if we set a huge expiry time so 604800 that pre-signed url would stop working when those temporary credentials for that roll also stopped working now it is possible to generate a pre-signed url from the console UI this is a relatively recent change from the object if you click on the object actions drop down you can click share with the pre-signed url you have to set the same settings so what you want the expiry to be in this particular case let's say 60 minutes and then I can go ahead and click on create pre-signed url now that's automatically copied into my clipboard and I can go ahead and move to a different tab paste that in and we can open the console UI generated pre-signed url so that's just an alternative way of doing the same process that we just used the cloud shell for now that's everything that I wanted to demonstrate in this demo lesson about pre-signed url and don't worry we're going to be talking about these more later in the course as well as looking at some alternatives what we need to do to tidy up this lesson is go back to the AWS console move to the S3 console and then just go ahead and empty and delete the bucket that you created so select animals for live media click on empty type or paste in permanently delete and then confirm or once that's successfully empty click on exit the bucket should still be selected and then go ahead and click on delete delete confirm that with the name of the bucket and then go ahead and delete the bucket and at this point we've cleaned up the account and the resources back in the same state as they were at the start of the lesson so I hope you've enjoyed this brief demo lesson go ahead complete the video and when you're ready I look forward to you joining me in the next video.

    1. Welcome back and in this lesson I want to talk to you about S3 pre-signed URLs.

      Pre-signed URLs are a way that you can give another person or application access to an object inside an S3 bucket using your credentials in a safe and secure way.

      Let's have a look at how this works architecturally.

      To illustrate how pre-signed URLs work, let's use this example architecture, an S3 bucket which doesn't have any public access configured.

      So it's still in a default private configuration.

      This means that in order to access the bucket or any resources in it an IAM user such as IAM admin would have to authenticate to AWS and be authorized to access the resource.

      IAM admin would send credentials along with the access request, AWS would validate them at the time that the request is made and only then grant access to the object in S3.

      Now one issue that we have is that because the bucket is private only authenticated users are able to access it.

      Our masked man here has no way of providing authentication information to AWS because he doesn't have any and so any request that's unauthenticated would fail.

      Now if giving the mystery user access to S3 is an essential requirement for the business then there are three common solutions at this point and none of them are ideal.

      Number one is to give the mystery user an AWS identity.

      Number two is to give the mystery user some AWS credentials to use or number three is to make the bucket or the object public.

      Now none of these are ideal.

      If the user only needs short term access to the bucket and objects the effort of supplying an identity seems excessive.

      Just giving some credentials to the user appears on the surface to be a security risk and definitely if it's somebody else's credentials that's just bad practice.

      Making the bucket public for anything which has sensitive data in it also appears to be less than ideal.

      So one solution that AWS offers is to use pre-signed URLs and let's look at how this would work with an architecture example.

      IAM admin is an AWS identity with some access rights granted via a permissions policy.

      So IAM admin can make the request to S3 to generate a pre-signed URL.

      She would need to provide her security credentials, specify a bucket name, an object key and an expiry date and time as well as indicate how the object would be accessed.

      And S3 will create a pre-signed URL and return it.

      This URL will have encoded inside it the details that IAM admin provided.

      So which bucket is for which object is for it will be encoded with the fact that the IAM admin user generated it and it will be configured to expire at a certain date and time as requested by the IAM admin user.

      The URL could then be passed to our mystery user and he or she could use it to access a specific object in the specific S3 bucket up until the point at which it expires.

      When the pre-signed URL is used the holder of that URL is actually interacting with S3 as the person who generated it.

      So for that specific object in that specific bucket until the timer expires our masked man is actually IAM admin.

      Pre-signed URLs can be used for both downloads from S3 so get operations or uploads to S3 known as put operations.

      So this type of architecture might be useful for the animals for life remote workers if they don't have access to AWS accounts or accessing from a secure location and just need to upload one specific object.

      Now there's another type of architecture which commonly uses pre-signed URLs and I've mentioned this earlier in the course when I was talking about some benefits of S3.

      I want you to think about a traditional application architecture.

      On the left we have an application user with a laptop.

      We've got an application server in the public cloud in the middle which hosts the application and for example say let's say this is a video processing application for wildlife videos that the animals for life organization manages.

      We've learned that one of the real strengths of S3 is its ability to host the large media files and so the large wildlife video files have been migrated from the application server to a media S3 bucket.

      But by doing this we've introduced a problem.

      Previously the videos were hosted on the application server and it could control access to the video files.

      If we host them on an S3 bucket then either every user needs an AWS identity so an IAM user to access the videos or we need to make the videos public so that the user running the web application can download them into her browser and neither of those are ideal.

      Luckily pre-signed URLs offer a solution.

      With pre-signed URLs we can keep the bucket private.

      Then we can create an IAM user in our AWS account for this application.

      Remember when I talked about choosing between an IAM user or a role I said that if you could visualize how many of a certain thing that would be using an identity then it's likely to suit an IAM user.

      Well in this case we have one application and application service accounts are a common scenario where IAM users are used.

      So for this example we create an IAM user for the application.

      So when our application user interacts with a web application she makes a request and that's step one.

      The request might be for a page of information on a bushfire which is currently happening in Australia which has a video file associated with it.

      The application that's running on the server knows that it can directly return the information that the request is asking for but that the video is hosted on the private S3 bucket.

      So it initiates a request to S3 asking for it to generate a pre-signed URL for that particular video object using the permissions of the IAM user that the application is using.

      So IAM app one.

      The S3 service then creates a URL which has encoded within it the authentication information for the IAM app one user.

      The access to one object in the bucket for a short time limited basis maybe two to three hours.

      And the S3 service then returns that to the application server and through to the end user.

      The web application that's running on the user's laptop then uses this pre-signed URL to securely access the particular object that's stored on the media bucket.

      Pre-signed URLs are often used when you offload media into S3 or as part of serverless architectures where access to a private S3 bucket needs to be controlled and you don't want to run thick application servers to broke at that access.

      And we'll look at serverless architectures later in the course.

      For now I just want to be sure that you understand what pre-signed URLs do and in the next lesson which is a demo lesson you're going to get the chance to experiment yourself and generate a pre-signed URL.

      Pre-signed URLs can be used to access an object in a private S3 bucket with the access rights of the identity which generates them.

      They're time limited and they encode all of the authentication information needed inside.

      And they can be used to upload objects and download objects from S3.

      Now I want to show you this in a demo because it's far easier to actually do it.

      But before we do that I want to step through some exam power-ups.

      There are a couple of really interesting facts about generating pre-signed URLs which might help you out in some exam questions.

      Now first and this is a fairly odd behavior but you can create a pre-signed URL for an object that you have no access to.

      The only requirement for generating a pre-signed URL is that you specify a particular object and an expiry date and time.

      If you don't have access to that object you can still generate a pre-signed URL which also because it's linked to you will have no access to that object.

      So there aren't many use cases where this is applicable but you do need to be aware that it is possible to generate a pre-signed URL when you don't have any access.

      Now when you're using the URL, so when you utilize a URL and attempt to access an object, the permissions that you have access to match the identity that generated it.

      And it's important to understand that it matches the permissions that the identity has right now.

      So at the point when you use the URL, the URL has the same permissions as the identity that generated it has right now.

      So if you get an access denied error when you attempt to use a pre-signed URL to access an object, it could be that the identity that generated that URL never had access or it could be that it simply doesn't have access right now.

      And they're two very important nuances to understand about pre-signed URLs.

      So when you're using a URL to access an object, it matches the current permissions of the identity that generated that URL.

      Now that's fairly okay for an IM user.

      As I demonstrated on the previous example of the application, generally you would create an IM user for the application and this IM user would have fairly static permissions.

      So the permissions generally wouldn't change between when you created the URL and when your customer is using that URL within an application.

      But don't generate pre-signed URLs based on an IM role.

      You can in theory assume an IM role, which remember gives you temporary credentials, and then you can use those temporary credentials to generate a pre-signed URL.

      A pre-signed URL can have a much longer validity period than those temporary credentials.

      So those temporary credentials will generally expire well before a pre-signed URL does.

      So if you generate a pre-signed URL using a role and then those temporary credentials expire, those credentials are no longer valid.

      And so that URL will stop working.

      And so it's almost never a good idea to generate a pre-signed URL using an IM role.

      You should always use long-term identities.

      So generally an IM user.

      With that being said though, that is everything that I wanted to cover in this theory lesson.

      So go ahead, complete the video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back and in this demo lesson you're going to configure S3 cross-region replication.

      So that's a replication of objects from one bucket to another bucket in different AWS regions.

      And the scenario that we're going to be stepping through is where you as a systems engineer are looking to configure replication to allow disaster recovery of an S3 static website from one region to another AWS region.

      So the first thing that I'll need you to do is to make sure that you're logged in to the general AWS account and you'll need to have the Northern Virginia region selected.

      Assuming that's the case then go ahead and move to the S3 console because we're going to be creating our source and destination buckets.

      Now if you see any notifications about updates to the UI or any additional functionality then just go ahead and close that down.

      What we want you to do is to go ahead and click on create bucket.

      Now we want to keep these names simple so that we can distinguish between source and destination bucket.

      So for the source bucket we're going to start with source bucket and then we want you to use your initials in my case AC and then we want you to put a random number at the end and I'm going to attempt to use 1337.

      For region for the source bucket let's use US - East - 1 and then scroll down past the block public access settings scroll down past bucket versioning and default encryption and then just create the bucket.

      In my case it successfully created a bucket using this name and again because of the globally unique naming requirements of S3 buckets you need to make sure that you pick something different than me and different from other students.

      Now the scenario is that we're using this bucket to host a static S3 website so we need to enable that functionality.

      So go into source bucket, click on properties, scroll down all the way to the bottom and click on edit next to static website hosting and we need to enable static website hosting and for hosting type just make sure that you select host a static website.

      Now we're going to use index.html for both the index document and for the error document.

      So enter index.html in both of those boxes and once you've done that you can save those changes.

      Now so that this bucket can host a static website we need to go to permissions and then we need to edit the block public access settings so click on edit and uncheck block all public access and once you've done so click on save changes.

      You need to confirm that so follow the instructions and then confirm and this means that the bucket can be made public but in order to make it public we need to add a bucket policy.

      So to edit the bucket policy scroll down below the block public access settings and then edit the bucket policy.

      Now attached to this lesson is a demo files link.

      I'll need you to click that link which will download a zip file.

      Go ahead and extract that zip file which will create a folder and go inside that folder.

      Containment in this folder are all the files which you'll need for this demo and inside that folder is a file called bucket_policy.json.

      Go ahead and open that file so this is the file bucket_policy.

      This is a bucket policy which allows any principal to use the star tile card to use the S3 get object action on this particular ARN.

      Now this is a placeholder we need to update this placeholder so that it references any objects within our source bucket so I want you to copy this entire bucket policy into your clipboard, move back to the console, paste it in and once you've pasted it in go ahead and copy the bucket ARN for this bucket into your clipboard by clicking on this icon and then I want you to select the placeholder that you've just pasted in so straight after the first speech mark all the way up to before the forward slash so select this component of this placeholder ARN and paste it in the source bucket ARN so it should look like this.

      It references ARN colon, ALUS colon, S3 and then colon, colon, colon and then the source bucket so whatever you've called your source bucket it should reference this so don't use what's on my screen, use your specific source bucket ARN and then you should have a forward slash star on the end.

      That means that any anonymous or unauthenticated identity will be able to get any objects within this S3 bucket and that's what we want, we want this to be a public bucket.

      So click on save changes to commit those changes and now this bucket is public, you get the warning under permissions overview that this bucket is public and under the bucket name at the top it will say publicly accessible.

      Now once you've done that go back to the S3 console and we're going to create the destination bucket, we're going to follow exactly the same process so click on create bucket, this time we'll call it destination bucket and then again your initials and then ideally the same random number so in my case 1-3-3-7.

      Go ahead and click on the region drop down and instead of picking US-EAST-1 this time we're going to use US-WEST-1 Now to save us some time we're going to uncheck this block all public access while we're creating the bucket so that we don't need to do that afterwards and we'll need to acknowledge that by checking this box, scroll all the way down to the bottom and then click on create bucket, then we're going to go into destination bucket, select properties, move down to the bottom and we'll need to enable static website hosting so click edit, enable, choose the option to host a static website and then just like before we're going to use index.html for both the index document and the error document so enter index.html in both of those boxes and then save those changes.

      Then we'll need to go to permissions and edit the bucket policy so scroll down, click on edit for bucket policy you'll need to copy this template bucket policy into your clipboard, paste it into the policy box and again we need to replace this placeholder so copy the destination bucket ARN into your clipboard, select from after the speech mark through to before the forward slash paste that in and that will now reference any objects in the destination bucket and go ahead and click on save changes so now at this point we have the source and destination bucket, both of them are public, both of them are set to be static websites and neither of them have any objects inside that bucket.

      The next step is to enable cross region replication from the source bucket through to the destination bucket so to do that click on the source bucket, click on the management tab, scroll down and we need to configure a replication rule so click on create replication rule, we're told that replication requires versioning to be enabled for the source bucket and we're given a convenient button to click to enable versioning on this bucket and that's what we're going to do so click on enable bucket versioning and this means this bucket can now be the source of this replication rule for replication rule name, I'm going to call it static website dr for disaster recovery we want it to be enabled straight away so make sure the status is set to enabled now we can limit this replication rule to have a certain scope, we can filter it based on prefix or tags and this allows us to replicate only part of this bucket, in our case we want to replicate the entire bucket so we're going to select the option that this rule applies to all objects in the bucket so check this option now that we've configured the source, so we've configured the bucket name, the source region and chosen the scope of this replication rule now we need to decide on the destination, now for the destination we can use a bucket within the same AWS account or we can specify a bucket within another AWS account, if we choose another AWS account then we need to worry about object ownership we need to make sure that the objects which are replicated into the destination account have the correct ownership because by default they will be owned by our account because we will be creating them we can specify as part of the replication rule that the destination account owns those objects but because we're using the same account that doesn't apply in this scenario so we're going to choose a bucket within this account, click on browse s3 and we're going to select the destination bucket that we just created so select the destination bucket that you created in the previous step and then click on choose path you're going to be informed that as well as replication requiring versioning on the source bucket it also is required on the destination bucket and again you're presented with a convenient button to enable versioning on this destination bucket so click on enable bucket versioning, just to confirm versioning is required on both the source and destination bucket when you're using s3 replication so that's the destination configured, we've picked the destination bucket and enabled versioning, so scroll down next we need to give this replication rule the permissions that it needs to interact with different AWS resources so this rule needs the ability to read from the source bucket and write those objects into the destination bucket so we need to give this replication rule an IAM role which provides those permissions so click in the choose IAM role drop down and select create new role you could select an existing one if you already had one which was preconfigured but for this demo lesson we've done so we're going to create a new role now you can enable the ability of s3 replication to support objects which are encrypted with AWS KMS you do have to select that option explicitly though when you're creating a replication rule and in our case we're not going to be replicating any encrypted objects using sse-kms so we can leave this unchecked I've talked about this in the theory lesson but you can also change the storage class as part of the replication process so generally you're replicating objects from a primary location or a source bucket through to a destination bucket and this is often part of disaster recovery scenarios where you often want cheaper storage at the destination end and so the default is to maintain the storage class that's used in the source bucket but you can override that and change the storage class which is used when storing those replicated objects in the destination bucket now we're not going to set that in this demonstration but you do have the option to do so now these are all relatively new features of s3 replication so s3 replication has been a feature available at AWS for some time but over time they've evolved and enhanced the feature sets available to the product so you're able to set replication time control or RTC and this imposes an SLA on the replication process so by enabling it you ensure that 99.99% of new objects are replicated within 15 minutes and this feature also provides access to replication metrics and notifications but do note that this does come at an additional cost because s3 versioning is used on both the source and destination buckets if you do delete an object in a source bucket then by default that deletion is not replicated through to the destination so by default s3 replication does not replicate delete markers and I've talked about delete markers elsewhere in the course by selecting this option you do replicate delete markers and that means that deletions are replicated from the source through to the destination for now though we're not going to select any of these just go ahead and click on save after a few moments you'll be presented with this option so the default s3 replication historically didn't replicate any objects which existed in a bucket before you enabled replication you're now offered the ability to replicate existing objects within a bucket so if you wanted to do this if you had a bucket with objects that already existed then you could replicate those as part of starting this process now our bucket is empty so we won't do this anyway so go ahead and select no do not replicate existing objects and click submit and at this point replication is enabled between our source bucket and our destination bucket so let's give it a try what we're going to do is click at the top here next to the source bucket name and this will take us to the top level of the source bucket and we're going to upload some objects so go ahead and click on upload and then click on add files at this point go ahead and locate the folder which you downloaded and extracted earlier in this demo lesson so the demo files folder so go inside that folder and inside here as well as the bucket policy template which you used earlier you'll see two folders website and website 2 I want you to expand the website and select both the aotm.jpg object and the index.html object select both of those and make sure you use the ones from the website folder not the website 2 folder and then click on open once they're selected for upload you need to scroll down to the bottom and click upload and now upload both of those objects to our source bucket once that's complete we can close down that dialog then I want you to click on properties we're going to move down to the bottom and we're going to copy the bucket website endpoint into our clipboard and then open that in a new tab so this is the source bucket website endpoint and this opens animalsforlife.org animal.monthformarch and this is a picture of my cat Winky so let's go back to the S3 console move to the top go back to the main S3 console and then move in to the destination bucket now don't be concerned if you see something slightly different at this point the timetaping for replicating objects from source to destination can vary wildly if you don't enable the replication SLA option in our case because these objects are relatively small they should be replicated fairly quickly now at this point I want you to go ahead and pause this video and wait until you see two objects in your destination bucket don't be alarmed if these objects take five or ten minutes to appear just wait pause the video wait for those objects to appear and then you're good to continue you'll see that we have both of the objects which we added to the source bucket they're both now stored in the destination bucket and that's because they've been replicated using S3 replication what we can do is go to properties, move all the way down to the bottom copy the destination bucket website endpoint into our clipboard and open that in a new tab so this is the destination static website endpoint and again we see the same website animalsforlife.org animal.monthformarch again my cat Winky so next we're going to move back to the main S3 console so click on Amazon S3 and then move back to the source bucket click on upload again and we're going to replace the objects that we just added so click on add files and then expand the website to folder and go ahead and upload both of these objects so we have the same name O T N dot JPEG and index dot HTML go ahead and upload both of these and make sure you're uploading to the source bucket so upload those you'll need to confirm that that'll take a few moments to complete once you see that the upload was successful you can close down this dialogue and then you should still have the tab open to the source bucket if you go to that tab and refresh you'll see now that we have animalsforlife.org animal.monthforapril and this is my cat Truffles if we go immediately to the destination bucket and hit refresh on this bucket you might still see the picture of Winky if the image doesn't change and you keep refreshing it just give it a few minutes because without the replication SLA then the time taken to replicate from source to destination can vary significantly in my case it took about three or four minutes of refreshers but now we see that the destination bucket has been updated with the new image again this is a picture of my cat Truffles so this has been a very simple example of cross region replication at this point I've got everything that I wanted to do in this demo lesson so all we need to do is to clean up the account and return it to the same state as it was at the start of the demo so close down both of these tabs and just return to the main S3 console select the destination bucket and click empty confirm this by following the instructions and then confirm click exit with the bucket still selected, delete the bucket again you need to follow the instructions to confirm deletion of this bucket so go ahead and do that and then we need to follow the same process for the source bucket so select it and empty it, confirm that, close down this dialogue and then delete the bucket and again you need to confirm that, follow those instructions and delete bucket then we need to click on services, move to the IAM console because remember S3 replication created an IAM role that was used before that replication so click on roles, locate the role which starts with S3 C R R and it will have the name of the source bucket as part of that role name so go ahead and select the correct role, make sure that you select the one which has the name of your source bucket and then delete that role and you need to confirm that process and that will delete the role and at this point that's the account back in the same state as it was at the start of this demo lesson so I hope you've enjoyed this demo and a simple example of how to use S3 replication specifically in this case cross region replication from a source to a destination bucket at this point that's everything that I wanted to do, I'll be talking about S3 replication elsewhere in the course in other associate courses and at the professional level so don't worry, you'll get plenty of opportunities to explore this specific ease of functionality available as part of S3 but at this point that's everything I wanted to do in this demo lesson go ahead, complete the video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome back.

      In this lesson, I want to talk about S3 replication, the feature which allows you to configure the replication of objects between a source and destination S3 bucket.

      Now there are two types of replication supported by S3.

      The first type, which has been available for some time, is cross-region replication or CRR, and that allows the replication of objects from source buckets to one or more destination buckets in different AWS regions.

      The second type of replication announced more recently is same-region replication or SRR, which as the name suggests is the same process, but where both the source and destination buckets are in the same AWS region.

      Now the architecture for both types of replication is pretty simple to understand once you've seen it visually.

      It only differs depending on whether the buckets are in the same AWS accounts or different AWS accounts.

      Both types of replication support both, so the buckets could be in the same or different AWS accounts.

      In both cases, replication configuration is applied to the source bucket.

      The replication configuration configures S3 to replicate from the source bucket to a destination bucket, and it specifies a few important things.

      The first is logically the destination bucket to use as part of that replication, and another thing that's configured in the replication configuration is an IAM role to use for the replication process.

      The role is configured to allow the S3 service to assume it, so that's defined in its trust policy.

      The role's permissions policy gives it the permission to read objects on the source bucket and permissions to replicate those objects to the destination bucket.

      And this is how replication is configured between source and destination buckets, and of course that replication is encrypted.

      Now the configuration does define a few other items, but I'll talk about them on the next screen for now, and just focusing on this basic architecture.

      There is one crucial difference between replication which occurs in the same AWS accounts versus different AWS accounts.

      Inside one account, both S3 buckets are owned by the same AWS account, so they both trust that same AWS account that they're in.

      That means that they both trust IAM as a service, which means that they both trust the IAM role.

      For the same account, that means that the IAM role automatically has access to the source and the destination buckets as long as the role's permission policy grants the access.

      If you're configuring replication between different AWS accounts, though, that's nothing off.

      The destination bucket, because it's in a different AWS account, doesn't trust the source account or the role that's used to replicate the bucket contents.

      So in different accounts, remember that the role that's configured to perform the replication isn't by default trusted by the destination account because it's a separate AWS account.

      So if you're configuring this replication between different accounts, there's also a requirement to add a bucket policy on the destination bucket, which allows the role in the source account to replicate objects into it.

      So you're using a bucket policy, which is a resource policy, to define the role in a separate account can write or replicate objects into that bucket.

      Once this configuration is applied, so either the top configuration is the same account or the bottom, if it's different accounts, then S3 can perform the replication.

      Now let's quickly review some of the options available for replication configuration, so that might actually come in handy for you to know.

      The first important option is what to replicate.

      The default is to replicate an entire source bucket to a destination bucket, so all objects, all prefixes, and all packs.

      You can, though, choose a subset of objects, so you can create a rule that has a filter, and the filter can filter objects by prefix or tags or a combination of both, and that can define exactly what objects are replicated from the source to the destination.

      You can also select which storage class the objects in the destination bucket will use.

      Now the default is to use the same class, but you can pick a cheaper class if this is going to be a secondary copy of data.

      Remember when I talked about the storage classes that are available in S3, I talked about infrequent access or one-zone infrequent access classes, which could be used for secondary data.

      So with secondary data, you're able to tolerate a lower level of resiliency, so we could use one-zone infrequent access for the destination bucket objects, and we can do that because we've always got this primary copy in the source bucket, so we can achieve better economies by using a lower cost storage class in the destination.

      So remember this is the example, the default is to use the same storage class on the destination as is used on the source, but you can override that in the replication configuration.

      Now you can also define the ownership of the objects in the destination bucket.

      The default is that they will be owned by the same account as the source bucket.

      Now this is fine if both buckets are inside the same account.

      That will mean that objects in the destination bucket will be owned by the same as the source bucket, which is the same account, so that's all good.

      However, if the buckets are in different accounts, then by default the objects inside the destination bucket will be owned by the source bucket account, and that could mean when you end up in a situation where the destination account can't read those objects because they're owned by a different AWS account.

      So with this option you can override that and you can set it so that anything created in the destination bucket is owned by the destination account.

      And lastly there's an extra feature that can be enabled called replication time control or RTC, and this is a feature which adds a guaranteed 15-minute replication SLA onto this process.

      Now without this, it's a best efforts process, but RTC adds this SLA, it's a guaranteed level of predictability, and it even adds additional monitoring so you can see which objects are queued for replication.

      So this is something that you would tend to only use if you've got a really strict set of requirements from your business, and make sure that the destination bucket and source buckets are in sync as closely as possible.

      If you don't require this, if this is just performing backups or it's just for a personal project, or if the source and destination buckets aren't required to always be in sync within this 15-minute window, then it's probably not worth adding this feature.

      It's something to keep in mind and be aware of for the exam.

      If you do see any questions that mention 15 minutes for replication, then you know that you need this replication time control.

      Now there are some considerations that you should be aware of, especially for the exam.

      These will come up in the exam, so please pay attention and try to remember these points.

      The first thing is that by default replication isn't retroactive.

      You enable replication on a pair of buckets of source and destination, and only from that point onward are objects replicated from source to destination.

      So if you enable replication on a bucket which already has objects, those objects will not be replicated.

      And related to this, in order to enable replication on a bucket, both the source and destination bucket need to have versioning enabled.

      You'll be allowed to enable versioning as part of the process of enabling replication, but it is a requirement to have it on, so a bucket cannot be enabled for replication without versioning.

      Now you can use S3 batch replication to replicate existing objects, but this is something that you need to specifically configure.

      If you don't, just remember by default replication is not retroactive.

      Secondly, it is a one-way replication process only.

      Objects are replicated from the source to the destination.

      If you add objects manually in the destination, they will not be replicated back to the source.

      This is not a bi-directional replication.

      It's one-way only.

      Now more recently, AWS did add the feature which allows you to add bi-directional replication, but just be aware that this is an additional setting which you need to configure.

      By default, replication is one-way, and enabling bi-directional replication is something you need to specifically configure, so keep that in mind.

      Now in terms of what does get replicated from source to destination, replication is capable of handling objects which are unencrypted, so if you don't have any encryption on an object, it's capable of handling objects which are encrypted using S3 and S3, and it's even capable of handling objects which are encrypted using S3 and KMS, but this is an extra piece of configuration that you'll need to enable.

      So there's configuration and there's extra permissions which are required, because of course KMS is emolved.

      Now more recently, AWS did add the ability to replicate objects encrypted with SSE-C, so that's server-side encryption with custom-managed keys, but this is a relatively recent addition.

      Historically, SSE-C was incompatible with cross or same-region replication.

      Replication also requires that the owner of the source bucket needs permissions on the objects which will replicate, so in most cases, if you create a bucket in an account and you add those objects, then the owner of the object will be the source account, but if you grant cross-account access to a bucket, if you add a resource policy allowing all the AWS accounts to create objects in a bucket, it's possible that the source bucket account will not own some of those objects, and the style of replication can only replicate objects where the source account owns those objects.

      So keep that in mind, and the limitation is it will not replicate system events, so if any changes are made in the source bucket by lifecycle management, they will not be replicated to the destination bucket, so only user events are replicated, and in addition to that, it can't replicate any objects inside a bucket that are using the Belysia or Glacia Deep Archive storage classes.

      Now that makes sense, because Glacia and Belysia Deep Archive, while they are shown as being inside an S3 bucket, you need to conceptually think of them as separate storage products, so they cannot be replicated using this process.

      And then lastly, it's important to understand that by default, deletes are not replicated between buckets, so the adding of a delete marker, which is how object deletions are handled for a version-enabled bucket, by default, these delete markers are not replicated.

      Now you can enable that, but you need to be aware that by default, this isn't enabled.

      So one of the important things I need to make sure you're aware of in terms of replication is why you would use replication.

      What are some of the scenarios that you'll use replication for?

      So for the same region replication specifically, you might use this process for log aggregation, so if you've got multiple different S3 buckets which store logs for different systems, then you could use this to aggregate those logs into a single S3 bucket.

      You might want to use the same region replication to configure some sort of synchronization between production and test accounts.

      Maybe you want to replicate data from prod to test periodically, or maybe you want to replicate some testing data into your prod account.

      This can be configured in either direction, but a very common use case for same region replication is this replication between different AWS accounts, different functions, so prod and test, or different functional teams within your business.

      You might want to use same region replication to implement resilience if you have strict sovereignty requirements.

      So there are companies in certain sectors which cannot have data leaving a specific AWS region because of sovereignty requirements, so you can have same region replication replicating between different buckets and different accounts, and then you have this account isolation for that data.

      So having a separate account with separate logins isolated to make an audit team or a security team replicates it into that account, it provides this account level isolation.

      Obviously, if you don't have those sovereignty requirements, then you can use cross region replication and use replication to implement global resilience improvements, so you can have backups of your data copied to different AWS regions to cope with large scale failure.

      You can also replicate data into different regions to reduce latency.

      So if you have, for example, a web application or your application loads data, then obviously it might be latency sensitive, so you can replicate data from one AWS region to another, so the customers in that remote region can access the bucket that's closest to them, and that reduces latency generally gives them better performance.

      Now that is everything I want to cover in this video, so go ahead and complete the video, and when you're ready, I look forward to you joining me in the next.

  5. Aug 2024
    1. Welcome back and in this lesson I want to talk about S3 lifecycle configuration.

      Now you can create lifecycle rules on S3 buckets which can automatically transition or expire objects in the bucket.

      There are great ways to optimize the cost for larger S3 buckets and so they're an important skill to have as a solutions architect, developer or operational engineer.

      Let's jump in and explore exactly what features lifecycle configuration adds to S3 buckets.

      A lifecycle configuration at its foundation is a set of rules which apply to a bucket.

      These rules consist of actions which apply based on criteria.

      So do X if Y is true.

      These rules can apply to a whole bucket or they can apply to groups of objects in that bucket defined by prefix or tags.

      The actions which are applied are one of two types.

      Transition actions which change the storage class of whichever object or objects are affected.

      So an example of this is that you could transition objects from S3 standard to say S3 infrequent access after 30 days.

      And you do this if you're sure that the objects will be more infrequently accessed after that initial 30 day period.

      You could then maybe transition the objects from infrequent access to S3 Glacier Depardive after 90 days.

      But you're certain that they will be rarely if ever accessed after that point.

      Now the other type of rule are expiration actions which can delete whatever object or object versions are affected.

      So you might want to expire objects or versions entirely after a certain time period.

      And this can be useful to keep buckets tidy.

      Now both of these can work on versions as well if you have a version enabled bucket.

      But this can become complex and so it's something that you need to carefully plan before implementing.

      Life cycle configurations offer a way to automate the deletion of objects or object versions or change the storage class of objects to optimize costs over time.

      And the important thing to understand is that these rules aren't based on access.

      So you can't move objects between classes based on how frequently they're accessed.

      This is something which is a different feature intelligent tearing does on your behalf.

      Now you can combine these S3 life cycle actions to manage an object's complete life cycle.

      For example suppose that objects you create have a well defined life cycle.

      Initially the objects are frequently accessed over a period of 30 days.

      Then objects are infrequently accessed for up to 90 days.

      And after that the objects are no longer required.

      In this scenario you can create an S3 life cycle rule in which you specify the initial transition action to S3 intelligent tearing, S3 standard IA or even S3 one zone IA storage.

      Then you might have another transition action through to one of the glacier storage classes for archiving and then potentially even an expiration action when you're sure you no longer require the object.

      As you move the objects from one storage class to another you save on the storage costs.

      Now let's look visually at the life cycle process.

      So with life cycle configurations specifically the transitions between different storage classes you can think of this as a waterfall.

      So we start with all of the storage classes.

      So we've got standard infrequent access, intelligent tearing, one zone infrequent access, glacier instant retrieval, glacier flexible retrieval and then glacier deep archive.

      And it's important to understand that this transition process like a waterfall flows down.

      So glacier flexible retrieval can transition into glacier deep archive.

      Glacier instant retrieval can transition into glacier flexible and glacier deep archive.

      One zone infrequent access can transition into glacier flexible or glacier deep archive.

      Importantly not into glacier instant retrieval.

      This is one of the exceptions you need to be aware of.

      Intelligent tearing this one can transition into one zone infrequent access, glacier instant, glacier flexible and glacier deep archive.

      Then we've got standard infrequent access which can transition into more classes still.

      So intelligent tearing, one zone infrequent access, glacier instant, glacier flexible and glacier deep archive.

      And then finally we've got S3 standard and this can transition into all of the other classes.

      So transitioning flows down like a waterfall but generally you won't go directly.

      Most of the life cycle configuration that I've been exposed to operates with multiple different stages as the access patterns of objects change over time.

      If you do have any use cases though you can go directly between most of the different classes.

      Whichever storage classes you do make use of transition can't happen in an upward direction only down.

      And then finally there are some restrictions or considerations that you need to be aware of.

      Firstly be careful when transitioning smaller objects from standard through to infrequent access, intelligent tearing or one zone infrequent access.

      This is because of the minimums on those classes.

      For larger collections of smaller objects you can end up with equal or more costs because of the minimums of these different storage classes.

      Additionally this is important to understand for the exam.

      There's a 30 day minimum period where an object needs to remain on S3 standard before then moving into infrequent access or one zone infrequent access.

      So you can store objects when you upload them directly into standard IA and one zone IA and that's okay.

      But if you first store that object in the S3 standard and then you want to transition that into standard IA or one zone IA then the object needs to have been in S3 standard for 30 days before you can life cycle transition that into either of those infrequent access tiers.

      That's really important to understand.

      You can always directly adjust the storage class of an object for the CLI or console UI.

      But when you're using life cycle configuration an object needs to be in S3 standard for 30 days before it can be transitioned into standard infrequent access or one zone infrequent access.

      And then finally this one is a little bit more obscure still.

      If you want to create a single rule which transitions objects from standard through to infrequent access or one zone infrequent access you have to wait an additional 30 days before then transitioning those objects through to any of the glacier classes.

      You can't have a single rule which moves something from S3 standard through to infrequent access classes and then soon in 30 days moves those same objects through to the glacier classes.

      With a single rule the object has to be within standard infrequent access or one zone infrequent access for 30 days before that same rule can then move those objects through to any of the glacier classes.

      You can have two rules which do that process without that 30 day gap but a single rule there has to be a 30 day period where the object is in the infrequent access tiers before moving into the glacier tiers.

      Now depending on the course that you're watching there might be a demo coming up elsewhere in the course where you'll use life cycle rules.

      If this doesn't apply to the course that you're on don't be alarmed if that isn't a demo.

      In any case that's all I wanted to cover in this theory lesson so go ahead and complete the video and when you're ready we can move on to the next.

    1. Welcome back, this is part two of this lesson.

      We're going to continue immediately from the end of part one.

      So let's get started.

      Now let's talk about the first of the Glacier Storage Classes, S3 Glacier Instant Retrieval.

      If I had to summarize this storage class, it's like S3 standard in frequent access, except it offers cheaper storage, more expensive retrieval costs, and longer minimums.

      Standard IA is designed for when you need data instantly, but not very often, say once a month.

      Glacier Instant Retrieval extends this, so data where you still want instant retrieval, but where you might only access it say once every quarter.

      In line with this, it has a minimum storage duration charge of 90 days versus the 30 days of standard in frequent access.

      This class is the next step along the path of access frequency, as the access frequency of objects decrease, you can move them gradually from standard, then to standard in frequent access, and then to Glacier Instant Retrieval.

      The important thing to remember about this specific S3 Glacier class is that you still have instant access to your data.

      There's no retrieval process required, you can still use it like S3 standard and S3 standard in frequent access.

      It's just that it costs you more if you need to access the data, but less if you don't.

      Now let's move on to the next type of S3 Glacier Storage Class.

      And the next one I want to talk about is S3 Glacier Flexible Retrieval, and this storage class was formally known as S3 Glacier.

      The name was changed when the previously discussed Instant Retrieval class was added to the lineup of storage classes available within S3.

      So Glacier Flexible Retrieval has the same three availability zone architecture as S3 standard and S3 standard in frequent access.

      It has the same durability characteristics of 11-9s, and at the time of creating this lesson, S3 Glacier Flexible Retrieval has a storage cost which is about one-sixth of the cost of S3 standard.

      So it's really cost effective, but there are some serious trade-offs which you have to accept in order to make use of it.

      For the exam, it's these trade-offs which you need to be fully aware of.

      Conceptually, I want you to think of objects stored with the Glacier Flexible Retrieval class as cold objects.

      They aren't warm, they aren't ready for use, and this will form a good knowledge anchor for the exam.

      Now because they're cold, they aren't immediately available, they can't be made public.

      Well, you can see these objects within an S3 bucket, they're now just a pointer to that object.

      To get access to them, you need to perform a retrieval process.

      That's a specific operation, a job which needs to be run to gain access to the objects.

      Now you pay for this retrieval process.

      When you retrieve objects from S3 Glacier Flexible Retrieval, they're stored in the S3 standard in frequent access storage class on a temporary basis.

      You access them and then they're removed.

      You can retrieve them permanently by changing the class back to one of the S3 ones, but this is a different process.

      Now retrieval jobs come in three different types.

      We have expedited, which generally results in data being available within one to five minutes, and this is the most expensive.

      We've got standard where data is usually accessible in three to five hours, and then a low cost bulk option where data is available in between five and 12 hours.

      So the faster the job type, the more expensive.

      Now this means that S3 Glacier Flexible Retrieval has a first byte latency of minutes or hours, and that's really important to know for the exam.

      So while it's really cheap, you have to be able to tolerate, you can't make the objects public anymore, either in the bucket or using static website hosting, and two, when you do access the objects, it's not an immediate process.

      So you can see the object metadata in the bucket, but the data itself is in chilled storage, and you need to retrieve that data in order to access it.

      Now S3 Glacier Flexible Retrieval has some other limits, so a 40 kb minimum available size and a 90 day minimum available duration.

      For the exam, Glacier Flexible Retrieval is for situations where you need to store archival data where frequent or real-time access isn't needed.

      For example, yearly access, and you're OK with minutes to hours for retrieval operations.

      So it's one of the cheapest forms of storage in S3, as long as you can tolerate the characteristics of the storage class, but it's not the cheapest form of storage.

      That honor goes to S3 Glacier Deep Archive.

      Now S3 Glacier Deep Archive is much cheaper than the storage class we were just discussing.

      In exchange for that, there are even more restrictions which you need to be able to tolerate.

      Conceptually, where S3 Glacier Flexible Retrieval, which data in a chilled state, Glacier Deep Archive is data in a frozen state.

      Objects have minimum, so 40 kb minimum available size and 180 day minimum available duration.

      Like Glacier Flexible Retrieval, objects cannot be made publicly accessible.

      Access to the data requires a retrieval job.

      Just like Glacier Flexible Retrieval, the jobs temporarily restore to S3 standard and frequent access, but those retrieval jobs take longer.

      Standard is 12 hours and bulk is up to 48 hours, so this is much longer than Glacier Flexible Retrieval, and that's the compromise that you agree to.

      The storage is a lot cheaper in exchange for much longer restore times.

      Glacier Deep Archive should be used for data which is archival, which rarely, if ever, needs to be accessed, and where hours or days is tolerable for the retrieval process.

      So it's not really suited to primary system backups because of this restore time.

      It's more suited for secondary long-term archival backups or data which comes under legal or regulatory requirements in terms of retention length.

      Now this being said, there's one final type of storage class which I want to cover, and that's intelligent tearing.

      Now intelligent tearing is different from all the other storage classes which I've talked about.

      It's actually the storage class which contains five different storage tiers.

      With intelligent tearing, when you move objects into this class, there are a range of ways that an object can be stored.

      It can be stored within a frequent access tier or an infrequent access tier, or for objects which are accessed even less frequently, there's an archive instant access, archive access, or deep archived set of tiers.

      You can think of the frequent access tier like S3 standard and the infrequent access tier like S3 standard infrequent access, and the archive tiers are the same price of performance as S3, Glacier, instant retrieval, and flexible retrieval.

      And the deep archive tier is the same price of performance as Glacier Deep Archive.

      Now unlike the other S3 storage classes, you don't have to worry about moving objects between tiers.

      With intelligent tearing, the intelligent tearing system does this for you.

      Let's say that we have an object, say a picture of whiskers which is initially kind of popular and then not popular, and then it goes super viral.

      Well if you store this object using the intelligent tearing storage class, it would monitor the usage of the object.

      When the object is in regular use, it would stay within the frequent access tier and would have the same costs as S3 standard.

      If the object isn't accessed for 30 days, then it would be moved automatically into the infrequent tier where it would stay while being stored at a lower rate.

      Now at this stage you could also add configuration, so based on a bucket, prefix or object tag, any objects which are accessed less frequently can be moved into the three archive tiers.

      Now there's a 90 day minimum for archive instant access, and this is fully automatic.

      Think of this as a cheaper version of infrequent access for objects which are accessed even less frequently.

      Crucially this tier, so archive instant access, still gives you access to the data automatically as and when you need it, just like infrequent access.

      In addition to this, there are two more entirely optional tiers, archive access and deep archive.

      And these can be configured so that objects move into them when they haven't been accessed for 98 to 270 days for archive access, or 180 through to 730 days for deep archive.

      Now these are entirely optional, and it's worth mentioning that when objects are moved into these tiers, getting them back isn't immediate.

      There's a retrieval time to bring them back, so only use these tiers when your application can tolerate asynchronous access patterns.

      So archive instant access requires no application or system changes, it's just another tier for less frequently accessed objects with a lower cost.

      Archive access and deep archive changes things, your applications must support these tiers because retrieving objects requires specific API calls.

      Now if objects do stay in infrequent access or archive instant access, when the objects become super viral in access, these will be moved back to frequent access automatically with no retrieval charges.

      Intelligent tiering has a monitoring and automation cost per 1000 objects instead of the retrieval cost.

      So essentially the system manages the movement of data between these tiers automatically without any penalty for this management fee.

      The cost of the tiers are the same as the base S3 tiers, standard and infrequent access, there's just the management fee on top.

      So it's more flexible than S3 standard and S3 infrequent access, but it's more expensive because of the management fee.

      Now intelligent tiering is designed for long-lived data where the usage is...

      [Sounds of S3 storage] Changing or unknown, if the usage is static either frequently accessed or infrequently accessed, then you're better using the direct S3 storage class, either standard or infrequent access.

      Intelligent tiering is only good if you have data where the pattern changes or you don't know it.

      Now with that being said, that's all of the S3 storage classes which I want to cover.

      That's at least enough technical information and context which you'll need for the exam and to get started in the real world.

      So go ahead and complete the video and when you're ready, I look forward to you joining me in the next.

    1. Welcome back and in this video I want to talk about S3 bucket keys which are a way to help S3 scale and reduce costs when using KMS encryption.

      Let's jump in and take a look.

      So let's look at a pretty typical architecture.

      We have S3 in the middle, we have KMS on the right and inside we have a KMS key, the default S3 service key for this region which is named AWS/S3.

      Then on the left we have a user Bob who's looking to upload some objects to this S3 bucket using KMS encryption.

      Within S3 when you use KMS each object which is put into a bucket uses a unique data encryption key or DEK.

      So let's have a look at how that works.

      So when Bob begins his first PUT operation when the object is arriving in the bucket a call is made to KMS which uses the KMS key to generate a data encryption key unique to this object.

      The object is encrypted and then the object and the unique data encryption key are stored side by side on S3.

      Each object stored on S3 uses a unique data encryption key which is a single call to KMS to generate that data encryption key.

      This means that for every single object that Bob uploads it needs a single unique call to KMS to generate a data encryption key to return that data encryption key to S3, use that key to encrypt the object and then store the two side by side.

      On screen we have three individual PUTs.

      But imagine if this was 30 or 300 or 300,000 every second.

      This presents us with a serious problem.

      KMS has a cost.

      It means that using SSE-KMS carries an ever increasing cost which goes up based on the number of objects that you put into an S3 bucket.

      And perhaps more of a problem is that there are throttling issues.

      The generated data encryption key operation can only be run either 5,500, 10,000 or 50,000 times per second and this is shared across regions.

      Now this exact number depends on which regions you use but this effectively places a limit on how often a single KMS key can be used to generate data encryption keys which limits the amount of PUTs that you can do to S3 every second.

      And this is where bucket keys improve the situation.

      So let's look at how.

      So with bucket keys the architecture changes a little.

      We have the same basic architecture but instead of the KMS key being used to generate each individual data encryption key, instead it's used to generate a time limited bucket key and conceptually this is given to the bucket.

      This is then used for a period of time to generate any data encryption keys within the bucket for individual object encryption operations.

      And this essentially offloads the work from KMS to S3.

      It reduces the number of KMS API calls so reduces the cost and increases scalability.

      Now it's worth noting that this is not retroactive.

      It only affects objects and the object encryption process after it's enabled on a bucket.

      So this is a great way that you can continue to use KMS for encryption with S3 but offload some of the intensive processing from KMS onto S3 reducing costs and improving scalability.

      Now there are some things that you do need to keep in mind when you're using S3 bucket keys.

      First, after you enable an S3 bucket key, if you're using CloudTrail to look at KMS logs, then those logs are going to show the bucket ARN instead of your object ARN.

      Now additionally, because you're offloading a lot of the work from KMS to S3, you're going to see fewer CloudTrail events for KMS in those logs.

      So that's logically offloading the work from KMS to S3 and instead of KMS keys being used to encrypt individual objects, they're used to generate the bucket key.

      And so you're going to see the bucket in the logs not the object.

      So keep that in mind.

      Book keys also work with same region replication and cross region replication.

      There are some nuances you need to keep in mind generally when S3 replicates an encrypted object.

      It generally preserves the encryption settings of that encrypted object.

      So the encrypted object in the destination bucket generally uses the same settings as the encrypted object in the source bucket.

      Now if you're replicating a plain text object, so something that's not encrypted and you're replicating that through to a destination bucket which uses default encryption or an S3 bucket key, then S3 encrypts that object on its way through to the destination with the destination bucket's configuration.

      And it's worth noting that this can result in e-tag changes between the source and the destination.

      Now make sure that I include a link attached to this video which details all of these nuanced features when you're using S3 bucket keys together with same or cross region replication.

      It's beyond the scope of this video, but it might be useful for the exam and the real world to be aware of these nuanced features and requirements as you're using the product.

      Now with that being said, that is everything that I wanted to cover in this video.

      So go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back and in this demo lesson I just want to give you the opportunity to gain some practical experience of how S3 handles encryption.

      So what we're going to do is create an S3 bucket and into that bucket we're going to put a number of objects and for each one we're going to utilize a different type of server-side encryption.

      So we'll be uploading one object using SSE-S3 so S3 managed encryption and then one object using SSE-KMS which will utilize KMS for key management.

      So once we've uploaded those objects we'll experiment with some permissions changes just to see how each of these different encryption types work.

      So let's get started.

      Now the first thing you'll need to check is that you're logged into the IAM admin user of the general AWS account and you need to have the Northern Virginia region selected.

      Then let's move to S3 so click in the services drop-down type S3 and then open that in a new tab and then click to move to the S3 console.

      Now I want you to go ahead and click on create bucket and then just create a bucket called catpicks and then put some random text on the end.

      You should use something different and something that's unique to you.

      Leave the region set as US East Northern Virginia scroll all the way down to the bottom and click on create bucket.

      Next change across to the key management service console that will either be in your history or you'll have to type it in the services drop-down.

      Once you hear go ahead and click on create key, pick symmetric key then expand advanced options, make sure KMS is selected as well as single region key.

      Click next.

      The alias we'll be using is catpicks, so type catpicks and click next.

      Don't select anything for define key administrative permissions.

      We'll not set any permissions on the key policy this time so just click next and then on this screen define key usage permissions just click next again without selecting anything.

      So the key policy that's going to be created for this KMS key only trusts the account so only trust the account user of this specific AWS account and that's what we want.

      So go ahead and click on finish.

      At this point move back across to the S3 console, go into the catpicks bucket that you just created and I'll let you to go ahead and download the file that's attached to this lesson and then extract that file and inside the resultant folder are some objects that you're going to be uploading to this S3 bucket.

      So go ahead and do that and then click on upload and files then locate the folder that you just extracted and go into it and you should see three images in that folder default -merlin.jpg, sse -kms -ginny.jpg and sse -s3 -dwees.jpg.

      So that's good so the first one that we want to upload need to do these one by one because we're going to be configuring the encryption type to use.

      So the first one is sse -s3 -dwees.jpg so select that and click on open, expand properties and then scroll down to serve aside encryption and it's here where you can specify to accept the bucket defaults by not specifying an encryption key or you can specify an encryption key.

      Now when you pick to specify an encryption key you're again offered the ability to use the bucket default settings for encryption or you can override bucket settings and choose between two different types either Amazon s3 key which is sse -s3 or AWS key management service which is sse -kms.

      Now for this upload we're going to use sse -s3 so Amazon s3 managed keys so select this option scroll all the way down to the bottom and click upload.

      Wait for this upload process to complete and then click on close.

      Now we're going to follow that same process again so click on upload again, add files.

      This time sse -kms -ginny.

      So select that click on open, expand properties and then scroll down to serve aside encryption then click specify an encryption key override bucket settings for default encryption and this time we're going to use sse -kms so select that and then select choose from your AWS kms keys and then you can either use the AWS managed key so this is the service default key so AWS forward slash s3 or you can choose your own kms key to use for this object.

      What I'm going to do first is select this AWS managed key so the default key for this service and scroll all the way down to the bottom and click on upload.

      Wait for this upload process to complete and then click on close so that will encrypt the object using this default s3 AWS managed kms key that's now inside kms.

      I wanted to do that just to demonstrate how it automatically creates it so now let's go ahead and re-upload this object so click on upload, add files, select sse -kms -ginny.jpg and click on open, scroll down, expand properties, scroll down again, for service side encryption select specify an encryption key, select override bucket settings for default encryption, pick AWS key management service key so sse -kms, select choose from your AWS kms keys, click in the drop down and then select captics that we created earlier.

      Once you've done that scroll all the way down to the bottom click on upload, wait for that process to complete and then click on close.

      Now at this point we've got two different objects in this bucket and we're going to open both of these.

      We're going to start with sse -s3 -dwees so let's click on it and then click on open and that works and then let's try the other object sse -kms -ginny so click on that and click on open and that also opens okay because IAM admin is a full administrator of the entire AWS account so that includes s3 and all the services including KMS.

      Next what we're going to do is apply a deny policy on the IAM admin user which prevents us from using KMS so we'll stay as being a full account administrator and a full s3 administrator but we're going to block off the KMS service entirely and I want to demonstrate exactly what that does to our ability to open these three objects.

      So click on services and either open the IAM console from the history or type it in the find services box and then click it.

      Once we're here click on users select your IAM admin user, click add permissions and then create inline policy, click on the JSON tab and then delete the skeleton template that's in this box and then paste in the contents of the deny kms.json file and this is contained in the folder you extracted from this lessons download link and it's also attached to this lesson.

      This is what it should look like the effect is to deny it denies any actions KMS call on star so any KMS actions on all resources so essentially this blocks off the entire KMS service for this user.

      So go ahead and click on review policy call it deny KMS and click on create policy and this now means that IAM admin can no longer access KMS so now if we go back to the s3 console go inside the capex bucket we should still be able to open sse-s3 dweez.jpeg object.

      If we click that click on open because this is encrypted using sse-s3 and this is completely internal to the s3 product we should have no problems owning this object because we have the permissions inside s3 to do anything in s3 but something different happens if we try to open the sse-kms-ginning object.

      Now just to explain what will occur when I click on this open link s3 will then have to liaise with KMS and get KMS to decrypt the data encryption key that encrypts this object so we need to retrieve the encrypted data encryption key for this object and request that KMS decrypts it.

      Now if that worked we'd get back the plain text version of that key and we would use it to decrypt this object and it would open up in a tab without any issues because we've got full rights over s3 we have permission to do almost all of that process but what we don't have permission to do is to actually get KMS to decrypt this encrypted data encryption key.

      We don't have that permission because we just added a deny policy to the IAM admin user and as we know by now deny allow deny deny always wins and explicit deny always overrules everything else.

      So when I click on open and s3 retrieves this encrypted data encryption key and gives it to KMS and says please decrypt this and give me the plain text back KMS is going to refuse and what we see when we do that is we get an access deny.

      So now we've implemented this role separation so even though we have full s3 admin rights so if I went back to this bucket and I clicked on the sse-kms-ginning file and deleted it I would be able to delete that object because I have full control over s3 but I can't open it because I've prevented us accessing KMS and that's how we implement role separation.

      So sse-kms is definitely the encryption type that you want to use if you've got these really extensive regulations or any security requirements around key control.

      So let's go ahead and just remove that restriction so just go back to the IAM console.

      I just want to do this before we forget and have problems later in the course.

      Click on users click on IAM admin check the box next to deny KMS and then click remove and confirm that removal and that will allow us to access KMS again.

      We can verify that by moving to the KMS console and we can bring up this list which proves that we've got some access to KMS again so that's good.

      Now if we just go ahead and click on the AWS managed keys option on the left here this is where you'll be able to see this default encryption key that's used when you upload an object using sse-kms-incription but don't pick a particular key so this is now the default.

      Now if we open this because it's an AWS managed key we don't have the ability to set any key rotation we can see the key policy here but we can't make any changes to it this is set by AWS when it creates it so that it only allows accesses from s3 so this is a fixed key policy but we can't control anything about this key.

      Now contrast that with the customer managed keys that we've got and if we go into cat pics this is the key that we created now we can edit the key policy we could switch to policy view and make changes we've got the ability to control the key rotation so if you face any exam questions where you need to fully manage the keys that are used as part of the s3 encryption process then you've got to use sse-kms.

      Now if we just return to the s3 console there's just one more thing that I want to demonstrate go into the cat pics bucket again click on properties locate default encryption and then click on edit and this is where you get the option to specify the default encryption to use for this bucket.

      Now again this isn't a restriction this does not prevent anyone uploading objects to the bucket using a different type of encryption all it does is specify what the default is if the upload itself does not specify an encryption method so we could select Amazon s3 key which is sse-s3 and you might also see this referred to elsewhere as AES 256 it's also known by that name but we could also select the AWS key management service key known as sse-kms and this is where we can either choose to use the default key or pick a customer managed key that we want to use as the default for the bucket.

      So let's just demonstrate that go ahead and select the cat pics key to use for this bucket then scroll down and click on save changes and that will set the defaults for the bucket and we can demonstrate that let's go ahead and click on the objects tab and we're going to upload a new object to this bucket so click on upload add files then go ahead and select the default hyphen Merlin object and click open scroll down click on upload and even though we didn't pick a particular encryption method for this object it's going to use the default settings that we picked for the bucket and now we can see that default hyphen Merlin.jpg object has been uploaded so if we open up default hyphen Merlin we can see it's using sse-kms as the service side encryption type and it's using the KMS key that we set in the default encryption settings on the bucket.

      Okay well that's everything I wanted to cover in this demo lesson so let's just tidy up to make sure that we don't experience any charges go back to the Amazon S3 console select the bucket that you've created and then click on empty you need to confirm to empty that bucket once that process is completed and the bucket's emptied and then follow that same process but this time click on delete to delete the bucket from your AWS account click on key management service and we'll just mark the key that were created for deletion so select customer managed keys select cat pics click on key actions schedule key deletion set this to 7 which is the minimum check the box and click on schedule deletion with that being said though that's everything I wanted to cover in this demonstration I hope it's been fun and useful go ahead mark this video is complete and when you're ready I'll see you in the next video.

    1. Welcome back and in this lesson I want to talk about S3 encryption.

      Now we're going to be focusing on server-side encryption known as SSE, which I will be coaching on client-side encryption and how that's different.

      Now we've got a lot to get through so let's jump in and get started.

      Now before we start there's one common misconception which I want to fix right away, and that's that buckets aren't encrypted, objects are.

      You don't define encryption at the bucket level.

      There's something called bucket default encryption, but that's different and I'll cover that elsewhere in the course.

      For now, understand that you define encryption at the object level, and each object in a bucket might be using different encryption settings.

      Now before we talk about the ways that S3 natively handles encryption for objects, I think it's useful to just review the two main architectures of encryption which can be used with the product.

      There's client-side encryption and server-side encryption, and both of these refer to what method is used for encryption at rest, and this controls how objects are encrypted as they're written to disk.

      It's a method of ensuring that even if somebody were to get the physical disks from AWS which your data is on, they would need something else, a type of key to access that data.

      So visually this is how a transaction between a group of users or an application and S3 looks like.

      The users of the application on the left are loading data to an S3 endpoint for a specific bucket which gets stored on S3's base storage hardware.

      Now it's a simplistic overview, but for this lesson it's enough.

      I want to illustrate the difference between client-side encryption and server-side encryption.

      So on the top we have client-side encryption, and on the bottom we have server-side encryption.

      Now this is a really, really important point which often confuses students.

      What I'm talking about in this lesson is encryption at rest, so how data is stored on disk in an encrypted way.

      Both of these methods also use encryption in transit between the user-side and S3.

      So this is an encrypted tunnel which means that you can't see the raw data inside the tunnel.

      It's encrypted.

      So ignoring any S3 encryption, ignoring how data is encrypted as it's written to disk, data transferred to S3 and from S3 is generally encrypted in transit.

      Now there are exceptions, but use this as your default and I'll cover those exceptions elsewhere in the course.

      So in this lesson when we're talking about S3 encryption, we're focusing on encryption at rest and not encryption in transit, which happens anyway.

      Now the difference between client-side encryption and server-side encryption is pretty simple to understand when you see it visually.

      With client-side encryption, the objects being uploaded are encrypted by the client before they ever leave, and this means that the data is ciphertexted the entire time.

      From AWS's perspective, the data is received in a scrambled form and then stored in a scrambled form.

      AWS would have no opportunity to see the data in its plain text form.

      With server-side encryption known as SSE, it's slightly different.

      Here, even though the data is encrypted in transit using HTTPS, the objects themselves aren't initially encrypted, meaning that inside the tunnel, the data is in its original form.

      Let's assume it's animal images.

      So you could remove the HTTP encrypted tunnel somehow and the animal pictures would be in plain text.

      Now once the data hits S3, then it's encrypted by the S3 servers, which is why it's referred to as server-side encryption.

      So to high level, the differences are with client-side encryption, everything is yours to control.

      You take on all of the risks and you control everything, which is both good and bad.

      You take the original data, you are the only one who ever sees the plain text version of that data, you generate a key, you hold that key and you manage that key.

      You are responsible for recording which key is used for which object, and you perform the encryption process before it's uploaded to S3, and this consumes CPU capacity on whatever device is performing the encryption.

      You just use S3 for storage, nothing else.

      It isn't involved in the encryption process in any way, so you own and control the keys, the process and any tooling.

      So if your organization needs all of these, if you have real reasons that AWS cannot be involved in the process, then you need to use client-side encryption.

      Now with server-side encryption known as SSE, you allow S3 to handle some or all of that process, and this means there are parts that you need to trust S3 with.

      How much of that process you trust S3 with and how you want the process to occur and determine which type of server-side encryption you use as there are multiple types.

      Now AWS has recently made server-side encryption mandatory, and so you can no longer store objects in an unencrypted form on S3.

      You have to use encryption at rest.

      So let's break apart server-side encryption and review the differences between each of the various types.

      There are three types of server-side encryption available for S3 objects, and each is a trade-off of the usual things, trust, overhead, cost, resource consumption and more.

      So let's quickly step through them and look at how they work.

      The first is SSE-C, and this is server-side encryption with customer-provided keys.

      Now don't confuse this with client-side encryption because it's very different.

      The second is SSE-S3, which is server-side encryption with Amazon S3 managed keys, and this is the default.

      The last one is an enhancement on SSE-S3, which is SSE-KMS, and this is server-side encryption with KMS keys stored inside the AWS Key Management Service, known as KMS.

      Now the difference between all of these methods is what parts of the process you trust S3 with and how the encryption process and key management is handled.

      At a high level, there are two components to server-side encryption.

      First, the encryption and decryption process.

      This is the process where you take plain text, a key and an algorithm, and generate cyber text.

      It's also the reverse, so taking that cyber text and a key and using an algorithm to output plain text.

      Now this is symmetrical encryption, so the same key is used for both encryption and decryption.

      The second component is the generation and management of the cryptographic keys, which are used as part of the encryption and decryption processes.

      These three methods of server-side encryption, they handle these two components differently.

      Now let's look at how.

      Now before we do, again, I just want to stress that SSE is now mandatory on objects within S3 buckets.

      This process will occur, you cannot choose not to use it.

      The only thing that you can influence is how the process happens and what version of SSE is utilized.

      Now first, with SSE-C, the customer is responsible for the keys, and S3 manages the encryption and decryption processes.

      So the major change between client-side encryption and this is that S3 are handling the cryptographic operations.

      Now this might sound like a small thing, but if you're dealing with millions of objects and a high number of transactions, then the CPU capability required to do encryption can really add up.

      So you're essentially offloading the CPU requirements of this process to AWS, but you still need to generate and manage the key or keys.

      So when you put an object into S3 using this method, you provide the plain text object and an encryption key.

      Remember this object is encrypted in transit by HTTPS on its way to S3, so even though it's plain text right now, it's not visible to an external observer.

      When it arrives at S3, the object is encrypted and a hash of the key is tagged to the object and the key is destroyed.

      Now this hash is one way, it can't be used to generate a new key, but if a key is provided during decryption, the hash can identify if that specific key was used or not.

      So the object and this one-way hash are stored on disk, assistantly.

      Remember S3 doesn't have the key at this stage.

      To decrypt, you need to provide S3 with the request and the key used to encrypt the object.

      If it's correct, S3 decrypts the object, discards the key and returns the plain text.

      And again, returning the object is done over an encrypted HTTPS tunnel, so from the perspective of an observer, it's not visible.

      Now this method is interesting.

      You still have to manage your keys, which does come with a cost and some effort, but you also retain control of that process, which is good in some regulation-heavy environments.

      You also save on CPU requirements versus client-side encryption, because S3 performs encryption and decryption, meaning smaller devices don't need to consume resources for this process.

      But you need to trust that S3 will discard the keys after use, and there are some independent audits which prove what AWS does and doesn't do during this process.

      So you choose SSE-C when you absolutely need to manage your own keys, but are happy to allow S3 to perform the encryption and decryption processes.

      You would choose client-side encryption when you need to manage the keys and also the encryption and decryption processes, and you might do this if you never want AWS to have the ability to see your plain text data.

      So let's move on to the next type of server-side encryption, and the type I want to describe now is SSE-S3.

      And with this method, AWS handles both the encryption processes as well as the key generation and management.

      When putting an object into S3, you just provide the plain text data.

      When an object is uploaded to S3 using SSE-S3, it's encrypted by a key which is unique for every object, so S3 generates a key just for that object, and then it uses that key to encrypt that object.

      For extra safety, S3 has a key which it manages as part of the service.

      You don't get to influence this, you can't change any options on this key, nor do you get to pick it.

      It's handled end-to-end by S3.

      From your perspective, it isn't visible anywhere in the user interface, and it's rotated internally by S3 out of your visibility and control.

      This key is used to encrypt the per-object key, and then the original key is discarded.

      What we're left with is a ciphertext object and a ciphertext key, and both of these are persistently stored on disk.

      With this method, AWS take over the encryption process just as with SSE-C, but they also manage the keys on your behalf, which means even less admin overhead.

      The flip side with this method is that you have very little control over the keys used.

      The S3 key is outside of your control, and the keys used to encrypt and encrypt objects are also outside of your control.

      For most situations, SSE-S3 is a good default type of encryption which makes sense.

      It uses a strong algorithm, AES256, the data is encrypted at rest and the customer doesn't have any admin overhead to worry about, but it does present three major problems.

      Firstly, if you're in an environment which is strongly regulated, where you need to control the keys used and control access to the keys, then this isn't suitable.

      If you need to control rotation of keys, this isn't suitable.

      And then lastly, if you need role separation, this isn't suitable.

      What I mean by role separation is that a full S3 administrator, somebody who has full S3 permissions to configure the bucket and manage the objects, then he or she can also decrypt and view data.

      You can't stop an S3 full administrator from viewing data when using this type of server-side encryption.

      And in certain industry areas such as financial and medical, you might not be allowed to have this small and open access for service administrators.

      You might have certain groups within the business who can access the data but can't manage permissions, and you might have requirements for another SIS admin group who need to manage the infrastructure but can't be allowed to access data within objects.

      And with SSE-S3, this cannot be accomplished in a rigorous best practice way.

      And this is where the final type of server-side encryption comes in handy.

      The third type of server-side encryption is server-side encryption with AWS Key Management Service Keys, known as SSE-KMS.

      How this differs is that we're now involving an additional service, the Key Management Service, or KMS.

      Instead of S3 managing keys, this is now done via KMS.

      Specifically, S3 and KMS work together.

      You create a KMS key, or you can use the service default one, but the real power and flexibility comes from creating a customer-managed KMS key.

      It means this is created by you within KMS, it's managed by you, and it has isolated permissions, and I'll explain why this matters in a second.

      In addition, the key is fully configurable.

      Now this seems on the surface like a small change, but it's actually really significant in terms of the capabilities which it provides.

      When S3 wants to encrypt an object using SSE-KMS, it has to liaise with KMS and request a new data encryption key to be generated using the chosen KMS key.

      KMS delivers two versions of the same data encryption key, a plain text version and an encrypted or cipher text version.

      S3 then takes the plain text object and the plain text data encryption key and creates an encrypted or cipher text object, and then it immediately discards the plain text key, leaving only the cipher text version of that key and both of these are stored on S3 storage.

      So you're using the same overarching architecture, the per object encryption key, and the key which encrypts the per object key, but with this type of server-side encryption, so using SSE-KMS, KMS is generating the keys.

      Now KMS keys can only encrypt objects up to 4KB in size, so the KMS key is used to generate data encryption keys which don't have those limitations.

      It's important to understand that KMS doesn't store the data encryption keys, it only generates them and gives them to S3.

      But you do have control over the KMS key, the same control as you would with any other customer-managed KMS key.

      So in regulated industries, this alone is enough reason to consider SSE-KMS because it gives fine-grained control over the KMS key being used as well as its rotation.

      You also have logging and auditing on the KMS key itself, and with CloudTrail you'll be able to see any calls made against that key.

      But probably the best benefit provided by SSE-KMS is the role separation.

      To decrypt an object encrypted using SSE-KMS, you need access to the KMS key which was originally used.

      That KMS key is used to decrypt the encrypted copy of the data encryption key for that object which is stored along with that object.

      If you don't have access to KMS, you can't decrypt the data encryption key, so you can't decrypt the object, and so it follows that you can't access the object.

      Now what this means is that if we had an S3 administrator, and let's call him Phil, because we're using SSE-KMS, it means Phil as an S3 administrator does have full control over this bucket.

      But because Phil has been given no permissions on the specific KMS key, he can't read any objects.

      So he can administer the object as part of administering S3, but he can't see the data within those objects because he can't decrypt the data encryption key using the KMS key because he has no permissions on that KMS key.

      Now this is an example of role separation, something which is allowed using SSE-KMS versus not allowed using SSE-S3.

      With SSE-S3, Phil as an S3 administrator could administer and access the data inside objects.

      However, using SSE-KMS, we have the option to allow Phil to view data in objects or not, something which is controllable by granting permissions or not on specific KMS keys.

      So time for a quick summary before we finish this lesson, and it's really important that you understand these differences for any of the AWS exams.

      With client-side encryption, you handle the key management and the encryption and decryption processes.

      Use this if you need to control both of those and don't trust AWS and their regular audits.

      This method uses more resources to manage keys as well as resources for actually performing the encryption and decryption processes at scale.

      But it means AWS never see your objects in plain text form because you handle everything end to end.

      This generally means you either encrypt all objects in advance or use one of the client-side encryption SDKs within your application.

      Now please don't confuse client-side encryption with server-side encryption, specifically SSE-C.

      Client-side encryption isn't really anything to do with S3, it's not a form of S3 encryption, it's different.

      You can use client-side encryption and server-side encryption together, there's nothing preventing that.

      So now let's step through server-side encryption, and remember this is now on by default, it's mandatory.

      The only choice you have is which method of SSE to use.

      With SSE-C you manage the encryption keys, you can use the same key for everything, but that isn't recommended.

      Or you can use individual keys for every single object, the choice is yours.

      S3 accepts your choice of key and an object and it handles the encryption and decryption processes on your behalf.

      This means you need to trust S3 with the initial plain text object and trust it to discard and not store the encryption key.

      But in exchange S3 takes over the computationally heavy encryption and decryption processes.

      And also keep in mind that the data is transferred in a form where it's encrypted in transit using HTTBS.

      So nobody outside AWS will ever have exposure for plain text data in any way.

      SSE-S3 uses AES-256, I mention this because it's often the way exam questions test your knowledge.

      If you see AES-256, think SSE-S3.

      With SSE-S3, S3 handles the encryption keys and the encryption process.

      It's the default and it works well for most cases, but you have no real control over keys, permissions or rotation.

      And it also can't handle role separation, meaning S3 for admins can access the data within objects that they manage.

      Finally we have SSE-KMS which uses KMS and KMS keys which the service provides.

      You can control key rotation and permissions, it's similar in operation to SSE-S3, but it does allow role separation.

      So use this if your business has fairly rigid groups of people and compartmentalised sets of security.

      You can have S3 admins with no access to the data within objects.

      Now for all AWS exams make sure you understand the difference between client side and server side encryption.

      And then for server side encryption try and pitch scenarios where you would use each of the three types of server side encryption.

      Now that's everything I wanted to cover in this lesson about object encryption, specifically server side encryption.

      Go ahead and complete this lesson, but when you're ready I look forward to you joining me in the next.

    1. Welcome back and in this demo lesson I just wanted to give you a bit of practical exposure to KMS.

      And in this demo we're going to continue using the scenario that we've got the cat ruler trying to send encrypted battle plans to the robot general.

      Now to get started just make sure that you're logged in into the IAM admin user of the general AWS account, so the management account of the organisation.

      As always you'll need to have the Northern Virginia region selected and once you do go ahead and move across to the KMS console.

      So click in the search box at the top, type KMS and then open that in a new tab.

      And then click on create key because we'll be creating a KMS key.

      Now KMS allows the creation of symmetric or asymmetric keys and just to keep this demonstration simple we're going to demonstrate using a symmetric key.

      So make sure this option selected and then just to demonstrate some of these options just expand this and this is where you can set the key material origin.

      If you recall from the previous lesson I talked about how the physical backing key material could be generated by KMS or imported and this is where you select between those two options.

      Now I'm not going to talk about the custom key store at this point in the course.

      I'll talk about this in more detail later on in the course when I talk about cloud HSM which is a different product entirely.

      For now just select KMS and this will make KMS generate this physical backing key material that we'll be using to do our cryptographic operations.

      Now historically KMS was a single region service which means that keys created in the product could never leave the region that they were created in.

      More recently KMS has extended this functionality allowing you to create multi region keys.

      Now for the purpose of this demo lesson we're only going to be using a single region key which is the default.

      So make sure that single region key is selected and then we can continue to go ahead and click on next.

      Now in the previous lesson I mentioned how a key has a unique ID but also that we could create an alias which references the key.

      And that's what we can do here so I'm going to go ahead and create an alias and I'm going to call the alias cat robot all onward.

      So type in cat robot and click next.

      Now I discussed earlier how a KMS key has a key policy and a key policy is a resource policy that applies to the key.

      Now it's here where we can specify the key administrators for a particular key.

      There is a difference between identities that can manage a key and identities that can use a key for cryptographic operations like encrypt or decrypt.

      It's this point where we define who can manage this key.

      So go ahead and just check the box next to I am admin.

      So that will make sure that our I am admin user can administer this key.

      Once you do that scroll down there's a box here that allows administrators also to delete this key and that's a default so we'll leave that as is and click next.

      Now the previous set is where we defined who had admin permissions on the key.

      This stage lets us define who can use the key for cryptographic operations so encrypt and decrypt.

      To keep things simple we're going to define a key which adds the relevant entries to the key policy.

      So just check the box next to I am admin which is the user that we logged in as and then just scroll down.

      So if you wanted to add other AWS accounts here so that they had permission to use this key but for this demonstration we don't need to do that.

      So just click on next and this is the key policy that this wizard has created.

      So if I just scroll down it assigns the account level trust at the top so the account itself, the account user is allowed to perform any KMS column actions on this key.

      So that is the part of the policy of this statement here which means that this key will trust this account.

      It's this statement that defines the key administrators so that I am admin inside this account can create and describe and enable and list and all the other admin style permissions.

      Scroll down further still it's this statement that allows I am admin to perform encryption and decrypt and re-encrypt and generate data key and describe key actions against this key.

      So all the permissions that we define are inside the key policy.

      So this point go ahead click on finish and that will create the key as well as the alias that we use to reference this key.

      If we go into this key I'll just show you some of the options that are available.

      We'll be able to obviously edit the key policy and we can define key rotation.

      So by default key rotation for a customer managed key is switched off and we can enable it to rotate the key once every year.

      For an AWS managed key that is by default turned on and you can't switch it off and it also performs a rotation approximately once every year.

      So just click on AWS managed keys as we go through the course and start turning on encryption for various different services.

      You'll notice how each service the first time it uses encryption with KMS it creates an AWS managed key in this list.

      Now that's everything we need to do on the AWS side.

      Now we can start using this key to perform some cryptographic operations.

      So let's do that.

      Now at this point rather than using the local command line interface on your local machine we're going to be using Cloud Shell.

      This allows us to use the same set of instructions regardless of your local operating system.

      So to launch Cloud Shell click on this icon and this will take a few minutes but it will put you at the shell that's using your currently logged in user in order to gain permission.

      So any commands you run in the shell will be run as your currently logged in user.

      So the first thing that we'll be doing is to create a plain text battle plan.

      So this is the message that the cab ruler is going to be sending to the robot general.

      To generate that file we'll use echo and then space and then a speechmark and then a small message and the message is going to be find all the doggos and then a comma distract them with the yums.

      So find all the doggos, distract them with the yums and then a speechmark to close that off and then we'll redirect that to a file called battleplans.txt.

      And then presenter.

      Now the commands to interact with KMS from the command line are fairly long so what I'll do is paste it in and then I'll step through it line by line and explain exactly what it's doing.

      So first we need to encrypt the plain text battle plans and we want the result to be a cipher text document something that we can pass to the robot general which can't be intercepted en route and can be decrypted at the other end.

      So this is the command that we need to run and I just want to step through this line by line.

      The top part should be fairly obvious so we're running the AWS command line tools, the KMS module and using the encrypt action.

      So this specifies that we want to encrypt piece of data.

      This line specifies the alias that we want to use to encrypt this piece of data.

      You can either specify the key ID which uniquely identifies a key or you can specify an alias using the alias forward slash and then the name of the alias.

      In this case I've lept it to do that so this is using the alias that we created in the first part of this demo lesson.

      This line is where we're providing the plain text to KMS and instead of typing the plain text on the command line we're telling it to consult this file so battleplans.txt.

      Now the next line is telling the command line tools to output the result as text and it's going to be a text output with a number of different fields.

      The next line, double hyphen query, is telling the command line tools to select one particular field and that's the field cipher text blob and it's this field that contains the cipher text output from the KMS command.

      Now the output of any of these commands that interact with KMS is going to be a base64 encoded file so it's not going to be binary data, it's going to be base64 encoded.

      What we want to do is have our output being a binary encrypted file and so we need to take the result of this encryption command and pass it to a utility called base64 and that utility using this command line option will decode that base64 and place the result into a not_battleplans.enc file and this is going to be our result in cipher text.

      Now I know that command is relatively complex, KMS is not the easiest part of AWS to use from the command line but I did want to step you through line by line so you didn't know what each line achieved.

      Ok so let's go ahead and run this command, to do that we need to click on paste and then once that's pasted into a cloud shell press enter to run the command and the output not_battleplans will be our encrypted cipher text.

      So if I run a cat not_battleplans we get binary encrypted data so obviously anyone looking from the outside will just see scrambled data and won't understand what the message is.

      So now I'm going to clear the screen to make it a little bit easier to see and this is the encrypted cipher text file that we could transfer across to the robot general.

      So now we need to assume in this scenario that we're now the robot general and we're looking to decrypt this file.

      Ok so now I'm going to paste the next command for this lesson which is the decrypt command and I'll be stepping through line by line just explaining exactly what each line accomplishes.

      So this is the command that you use to decrypt the cipher text and give us the original plain text battle plans.

      So first this top line should be logical we're running the AWS command line tools with the KMS module and the decrypt command.

      We're passing in some cipher text so we use the command line option double hyphen cipher text blob and instead of pasting this on the command line we're giving it this file so not_battleplans.enc.

      We're again asking for the output to be in text.

      This will output some text with a number of different fields we're using the query field query for the plain text field and again the output will be base 64 encoded and so we're using the base 64 utility with the double hyphen decode to decode that back into its original form and store that into a file called decryptorplans.txt.

      So let's go ahead and run this so click paste and then press enter to run this command.

      This will decrypt cipher text and it will output decryptorplans.txt.

      And if we catch that document we'll see the original message.

      Find all the doggos, distract them with the yums and that's just been a really simple demonstration of using the KMS encrypt command and the KMS decrypt command.

      A couple of things I wanted to draw your attention to throughout the process.

      With the encrypt command we needed to pass in the key to use as well as the plain text and we got out the cipher text.

      With the decrypt command we don't need to specify the key, we only give the cipher text and assuming we have permissions on the KMS key so that we can use it to perform decrypt operations then we'll get the decryptor plain text and that's what's happened here.

      Now just to clear up from this lesson if you go back to the AWS console make sure you're in US East 1 so Northern Virginia and go back to the key management service console and we're just going to delete the KMS key that we created earlier in this lesson.

      So click on customer managed keys, select the KMS key that we created earlier, my case, cap robot then click on key actions and schedule key deletion.

      You need to enter a waiting period between 7 and 30 days since you want this cleared up as fast as possible going into 7, tick the box to confirm and then schedule delete.

      And that'll put the key into a pending deletion state and after 7 days it'll be entirely removed.

      And at that point we've cleared up all of the assets that we've used in this demo lesson so go ahead and complete the video and when you're ready join me in the next.

    1. Welcome to this video where I'm going to be talking about the key management service known as KMS.

      Now this product is used by many other services within AWS when they use encryption, so don't be surprised if you're watching this video in what seems like a pretty random place within the course.

      With that being said let's jump in and get started.

      Now KMS isn't all that complex as a product.

      Once you understand it it's pretty simple, but because of how much it's used by other AWS products and services it's essential that you do understand it for all the AWS exams.

      Now KMS is a regional and public service.

      Every region is isolated when using KMS.

      Think of it as a separate product.

      Now KMS is capable of some multi-region features but I'll be covering those in a separate dedicated video.

      It's a public service which means it occupies the AWS public zone and can be connected to from anywhere with access to this zone.

      Like any other AWS service though you will need permissions to access it.

      Now KMS as the name suggests manages keys.

      Specifically it lets you create, store and manage cryptographic keys.

      These are keys which can be used to convert plain text to ciphertext and vice versa.

      Now KMS is capable of handling both symmetric and asymmetric keys.

      And at this point you should understand what that means.

      Where symmetric keys are used, where public asymmetric are used, as well as private asymmetric.

      Just know that KMS is capable of operating with all of these different key architectures.

      Now KMS is also capable of performing cryptographic operations which includes, but is not limited to, encryption and encryption operations.

      And I'll be talking more about this later in this video.

      Now one of the foundational things to understand about KMS is that cryptographic keys never leave the product.

      KMS can create keys, keys can be imported, it manages keys, it can use these keys to perform operations but the keys themselves are locked inside the product.

      Its primary function is to ensure the keys never leave and held securely within the service.

      Now KMS also provides a FIPS 140-2 compliant service.

      This is a US security standard but try to memorize this.

      It's FIPS 140-2 level 2 to be specific.

      Again the level 2 part matters.

      It's often a key point of distinction between using KMS versus using something like cloud HSM which I'll be covering in detail elsewhere.

      Now some of KMS's features have achieved level 3 compliance but overall it's level 2.

      Again please do your best to remember this.

      It will come in handy for most of the AWS exams.

      Now before we continue, since this is an introduction video, unless I state otherwise, assume that I'm talking about symmetric keys.

      When I mention keys within this video, I'm going to be covering the advanced functionality of KMS in other videos including asymmetric keys but for this one I'm mainly focusing on its architecture and high level functions.

      So just assume I'm talking about symmetric keys from now on unless I indicate otherwise.

      Now the main type of key that KMS manages are known logically enough as KMS keys.

      You might see these referred to as CMKs or Customer Master Keys but that naming scheme has been superseded so they're now called KMS keys.

      These KMS keys are used by KMS within cryptographic operations.

      You can use them, applications can use them and other AWS services can use them.

      Now they're logical, think of them as a container for the actual physical key material and this is the data that really makes up the key.

      So a KMS key contains a key ID, a creation date, a key policy which is a resource policy, a description and a state of the key.

      Every KMS key is backed by physical key material, it's this data which is held by KMS and it's this material which is actually used to encrypt and decrypt things that you give to KMS.

      The physical key material can be generated by KMS or imported into KMS and this material contained inside a KMS key can be used to directly encrypt or decrypt data up to 4KB in size.

      Now this might sound like a pretty serious limitation.

      KMS keys are generally only used to work on small bits of data or to generate other keys and I'll be covering this at a high level later in this video.

      Let's look visually at how KMS works so far.

      So this is KMS and this is Ashley.

      Ashley's first interaction with KMS after picking a region is to create a KMS key.

      A KMS key is created with physical backing material and this key is stored within KMS in an encrypted form.

      Nothing in KMS is ever stored in plain text form persistently.

      It might exist in memory in plain text form but on disk it's encrypted.

      Now Ashley's next interaction with KMS might be to request that some data is encrypted.

      To do this she makes an encrypt call to KMS specifying the key to use and providing some data to encrypt.

      KMS accepts the data and assuming Ashley has permissions to use the key, it decrypts the KMS key then uses this key to encrypt the plain text data that Ashley supplied and then returns that data to Ashley.

      Notice how KMS is performing the cryptographic operations.

      Ashley is just providing data to KMS together with instructions and it's handling the operations internally.

      Logically at some point in the future Ashley will want to decrypt this same data so she calls a decrypt operation and she includes the data she wants to decrypt along with this operation.

      KMS doesn't need to be told which KMS key to use for the decrypt operation.

      That information is encoded into the cyber text of the data which Ashley wants to decrypt.

      The permissions to decrypt are separate from the permissions to encrypt and are also separate from permissions which allow the generation of keys but assuming Ashley has the required permissions for a decrypt operation using this specific KMS key, KMS decrypts the key and uses this to decrypt the cyber text provided by Ashley and returns this data back to Ashley in plain text form.

      Now again I want to stress at no point during this entire operation do the KMS keys leave the KMS product.

      At no point are the keys stored on the disk in plain text form and at each step Ashley needs permissions to perform the operations and each operation is different.

      KMS is very granular with permissions.

      You need individual permissions for various operations including encrypt and decrypt and you need permissions on given KMS keys in order to use those keys.

      Ashley could have permissions to generate keys and to use keys to encrypt and decrypt or she could have just one of those permissions.

      She might have permissions to encrypt data but not decrypt it or she might have permissions to manage KMS creating keys and setting permissions but not permissions to use keys to encrypt or decrypt data and this process is called "Rowl Separation".

      Now I mentioned at the start of this lesson that the KMS key can only operate cryptographically on data, which is a maximum of 4kb in size.

      Now that's true, so let's look at how KMS gets around this.

      Data encryption keys, also known as DEX or D-E-Ks, are another type of key which KMS can generate.

      They're generated using a KMS key, using the generate data key operation.

      This generates a data encryption key which can be used to encrypt and decrypt data which is more than 4kb in size.

      Data encryption keys are linked to the KMS key which created them, so KMS can tell that a specific data encryption key was created using a specific KMS key.

      But, and this is pretty much the most important thing about KMS and data encryption keys, KMS doesn't store the data encryption key in any way.

      It provides it to you or the service using KMS and then it discards it.

      The reason it discards it is that KMS doesn't actually do the encryption or decryption of data using data encryption keys.

      You do or the service using KMS performs those operations.

      So let's look at how this works.

      The data encryption key is generated, KMS provides you with two versions of that data encryption key.

      First, a plain text version of that key, something which can be used immediately to perform cryptographic operations.

      And second, a ciphertext or encrypted version of that same data encryption key.

      The data encryption key is encrypted using the KMS key that generated it.

      And in future, this encrypted data encryption key can be given back to KMS for it to be decrypted.

      Now the architecture is that you would generate a data encryption key immediately before you wanted to encrypt something.

      You would encrypt the data using the plain text version of the data encryption key and then once finished with that process, discard the plain text version of that data encryption key.

      That would leave you with the encrypted data and you would then store the encrypted data encryption key along with that encrypted data.

      Now a few key things about this architecture.

      KMS doesn't actually do the encryption or decryption on data larger than 4KB using data encryption keys.

      You do or the service using KMS does.

      KMS doesn't track the usage of data encryption keys.

      That's also you or the service using KMS.

      You could use the same data encryption key to encrypt 100 or a million files or you could request a new data encryption key for each of those million files.

      How you decide to do this is based on your exact requirements and of course AWS services will make this choice based on their requirements.

      By storing the encrypted data encryption key on disk with the encrypted data, you always have the correct data encryption key to use.

      But both the deck and the data are encrypted so administration is easy and security is maintained.

      When you're encrypting that data is simple.

      You pass the encrypted data encryption key back to KMS and ask for it to decrypt it using the same KMS key used to generate it.

      Then you use the decrypted data encryption key that KMS gives you back and decrypt the data with it and then you discard the decrypted data encryption key.

      Services such as S3 when using KMS generate a data encryption key for every single object.

      They encrypt the object and then discard the plain text version.

      As we move through the course I'll be talking in detail about how those services integrate with KMS for encryption services.

      Before we finish up with this lesson there are a few key concepts which I want to discuss.

      The one thing which is really important to grasp with KMS is that by default KMS keys are stored within the KMS service in that specific region.

      They never leave the region and they never leave the KMS service.

      You cannot extract a KMS key.

      Any interactions with a KMS key are done using the APIs available from KMS.

      Now this is the default but KMS does support multi-region keys where keys are replicated to other AWS regions.

      But I'll be covering that in a dedicated video if required for the course that you're studying.

      In KMS as a product keys are either AWS owned or customer owned.

      We're going to be dealing mainly with customer owned keys.

      AWS owned keys are a collection of KMS keys that an AWS service owns and manages for use in multiple AWS accounts.

      They operate in the background and you largely don't need to worry about them.

      If applicable for the course that you're studying I'll have a separate video on this.

      If not don't worry it's unimportant.

      Now in dealing with customer owned keys there are two types.

      AWS managed and customer managed and I'll be covering the specifics of these in a dedicated video.

      AWS managed keys are created automatically by AWS when you use a service such as S3 which integrates with KMS.

      Customer managed keys are created explicitly by the customer to use directly in an application or within an AWS service.

      Customer managed keys are more configurable.

      For example you can edit the key policy which means you could allow cross account access so that other AWS accounts can use your keys.

      AWS managed keys can't really be customized in this way.

      Both types of keys support rotation.

      Rotation is where physical backing materials are the data used to actually do cryptographic operations is changed.

      With AWS managed keys this can't be disabled.

      It's set to rotate approximately once per year.

      With customer managed keys rotation is optional.

      It's enabled by default and happens approximately once every year.

      A KMS key contains the backing key, the physical key material and all previous backing keys caused by rotation.

      It means that as a key is rotated data encrypted with all versions can still be decrypted.

      Now you can create aliases which are shortcuts to keys.

      So you might have an alias called my app one which points at a specific KMS key.

      That way KMS keys can be changed if needed.

      But be aware the aliases are also per region.

      You can create my app one in all regions but in each region it will point at a different key.

      Neither aliases or keys are global by default.

      Okay to finish up this KMS 101 lesson I want to talk at high level about permissions on KMS keys.

      Permissions on keys are controlled in a few ways.

      KMS is slightly different than other AWS services that you come across in terms of how keys are handled.

      Many services will always trust the account that they're contained in.

      Meaning if you grant access via an identity policy that access will be allowed unless there's an explicit deny.

      KMS is different.

      This account trust is explicitly added on a key policy or not.

      The starting point for KMS security is the key policy.

      This key policy is a type of resource policy like a bucket policy only on a key.

      Every KMS key has one and for custom managed keys you can change it.

      To reiterate this the reason the key policy is so important is that unlike other AWS services KMS has to explicitly be told that keys trust the AWS account that they're contained within.

      And this is what a key policy might look like.

      It means that the key will allow the account 11112222333 to manage it.

      This trust isn't automatic so be careful when updating it.

      You always need this type of key policy in place if you want to be able to grant access to a key using identity policies.

      The key doesn't trust the AWS account and this means that you would need to explicitly add any permissions on the key policy itself.

      Generally KMS is managed using this combination of key policies trusting the account and then using identity policies to let IAM users interact with the key.

      But in high security environments you might want to remove this account trust and insist on any key permissions being added inside the key policy.

      And a typical IAM permissions policy for KMS might look something like this which gives the holder of the policy the rights to use this key to encrypt or decrypt data.

      Inside KMS permissions are very granular and can be split based on function.

      You can be granted rights to create keys and manage keys but not to have permissions to perform cryptographic operations like encrypt or decrypt.

      This way your product administrators are given rights to access data encrypted by KMS which is a common requirement of many higher security environments.

      Now there's another way to interact with KMS using grants but I'll be covering this elsewhere in another video if needed.

      So that's everything I wanted to cover in this KMS introduction video.

      This video is going to form the foundation for others in this series depending on the topic that you're studying there might be no more videos or many more videos.

      Don't be worried it'll be the case.

      Now at this point that's everything I wanted to talk about though about KMS at this introductory level.

      Go ahead and complete the video and when you're ready I look forward to you joining me in the next.

    1. Welcome back.

      In this demo lesson, I just want to give you a really quick overview of the S3 performance improvement feature that we talked about in the previous lesson.

      Now, I want to make this really brief because it's not something that I can really demonstrate effectively because I have a relatively good internet connection.

      So I'm not going to be able to achieve the differences in performance that will effectively demonstrate this feature.

      But we can see how to enable it and then use an AWS tool to see exactly what benefits we can expect.

      So to enable it, we can move to the S3 console and just create a bucket.

      Now, you won't be able to use bucket names with periods in with this accelerator transfer tool.

      So this would not be a valid bucket name to use.

      It cannot have periods in the name.

      So I'm going to go ahead and create a test bucket.

      I'm going to call it test AC1279, which is something I don't believe I've used before.

      Now, you should go ahead and create a bucket yourself just to see where to enable this, but you can just follow along with what I'm doing.

      This won't be required at any point further down the line in the course.

      So if you just want to watch me in this demo, that's completely okay.

      So I'll create this bucket.

      I'll accept the rest of the defaults.

      Then I'll select the bucket and go to properties and it's right down at the bottom where you enable transfer acceleration.

      So this is an on/off feature.

      So if I select it, I can enable transfer acceleration or disable it.

      And when I enable it, it provides a new endpoint for this bucket to utilize the transfer acceleration feature.

      So that's important.

      You need to use this specific endpoint in order to get the benefit of these accelerator transfers.

      And this will resolve to an edge location that is highly performant wherever you're physically located in the world.

      So it's important to understand that you need to use this endpoint.

      So enable this on the bucket and click on save and that's all you need to do.

      That enables the feature.

      Now, as I mentioned at the start of this lesson, this is not the easiest feature to demonstrate, especially if you have a good internet connection.

      This is much better as a demonstration if I was on a suboptimal connection, which I'm not right now.

      But what you can do is click on the URL which is attached to this lesson, which opens a comparison tool.

      So what this is going to do now is it's going to compare the direct upload speeds that I can achieve from my local machine to this specific S3 bucket.

      And it's going to do so with and without transfer acceleration.

      So it will be giving you a comparison of exactly what speed you can expect to upload to a specific AWS region, such as US East 1, and then how that compares, uploading to that same region using accelerator transfer.

      So we can see already this is demonstrating a 144% faster upload speed from my location to Northern Virginia by using accelerator transfer.

      Now, I'm going to allow this to continue running while I continue talking, because what you'll see if you run this tool is a different set of results than what I'm seeing.

      You'll see different benefits in each region of using accelerator transfer, depending on your home location.

      So if you're located in San Francisco, for example, you probably won't see a great deal of difference between directly uploading and using accelerator transfer.

      But for more distant regions, you'll see a much more pronounced improvement.

      So if I just move down on this page and make these different regions a little bit easier to see, you'll note, for example, I achieve a much larger benefit in Oregon than I do in San Francisco.

      And my result for Dublin, which is even further away from my current location, is a yet higher benefit for using accelerator transfer.

      So the less optimal the network route is between your location and the region that's being tested, the better benefit you'll achieve by using accelerator transfer.

      Now, there are quite a lot of AWS regions, so I'm not going to let this test finish, but I do recommend if you are interested in S3 performance and this feature specifically, you should test this from your internet connection and allow this process to finish, because it will give you a really good indication of what performance you can expect to each of the AWS regions, and then how S3 accelerator transfer will improve that performance.

      Now, that is everything I wanted to cover in this lesson.

      I know it's been a brief demo lesson, and it isn't really a demo lesson where you're doing anything practically, but I did just want to supplement the previous lesson by giving you a visual example of how this feature can improve performance to S3.

      And I do hope you'll try this tool from your internet connection so you can see the benefit it provides from your location.

      With that being said, though, that is everything that I wanted to cover, so go ahead, complete this video, and when you're ready, I'll see you in the next.

    1. Welcome back, and this time we're going to cover a few performance optimization aspects of S3.

      If you recall from earlier in the course, this is the Animals For Life scenario.

      We have a head office in Brisbane, remote offices which consume services from the Brisbane office, and remote workers using potentially slower or less reliable services to access and upload data to and from the head office.

      So keep the scenario in mind as we step through some of the features that S3 offers to improve performance.

      It's not always about performance.

      It's often about performance and reliability combined.

      And this is especially relevant when we're talking about a distributed organization such as Animals For Life.

      So let's go ahead and review the features that S3 offers, which help us in this regard.

      Now, understanding the performance characteristics of S3 is essential as a solutions architect.

      We know from the Animals For Life scenario that remote workers need to upload large data sets and do so frequently.

      And we know that they're often on unreliable internet connections.

      Now, this is a concern because of the default way that S3 uploads occur.

      By default, when you upload an object to S3, it's uploaded as a single blob of data in a single stream.

      A file becomes an object, and it's uploaded using the put object API call and placed in a bucket.

      And this all happens as a single stream.

      Now, this method has its problems.

      While it is simple, it means that if a stream fails, the whole upload fails, and the only recovery from it is a full restart of the entire upload.

      If the upload fails at 4.5 GB of a 5 GB upload, that's 4.5 GB of data wasted and probably a significant amount of time.

      Remember, the data sets are being uploaded by remote workers over slow and potentially unreliable internet links.

      And this data is critical to the running of the organization.

      Any delay can be costly and potentially risky to animal welfare.

      When using this single put method, the speed and reliability of the upload will always be limited because of this single stream of data.

      If you've ever downloaded anything online, it's often already using multiple streams behind the scenes.

      There are many network-related reasons why even on a fast connection, one stream of data might not be optimal, especially if the transfer is occurring over long distances.

      In this type of situation, single stream transfers can often provide much slower speeds than both ends of that transfer are capable of.

      If I transfer you data with a single stream, it will often run much slower than my connection can do and your connection can do.

      Remember, when transferring data between two points, you're only ever going to experience the speed, which is the lowest of those two points, but often using single stream transfer, you don't even achieve that.

      Data transfer protocols such as BitTorrent have been developed in part to allow speedy distributed transfer of data.

      And these have been designed to address this very concern.

      Using data transfer with only a single stream is just a bad idea.

      Now, there is a limit within AWS if you utilize a single put upload, then you're limited to 5 GB of data as a maximum.

      But I would never trust a single put upload with anywhere near that amount of data.

      It's simply unreliable.

      But there is a solution to this.

      And that solution is multi-part upload.

      Multi-part upload improves the speed and reliability of uploads to S3.

      And it does this by breaking data up into individual parts.

      So we start with the original blob of data that we want to upload to S3, and we break this blob up into individual parts.

      Now, there is a minimum.

      The minimum size for using multi-part upload is 100 MB.

      So the minimum size for this original blob of data is 100 megabytes.

      You can't use multi-part upload if you're uploading data smaller than this.

      Now, my recommendation is that you start using this feature the second that you can.

      The most AWS tooling will automatically use it as soon as it becomes available, which is at this 100 MB lower threshold.

      There are almost no situations where a single put upload is worth it when you get above 100 MB.

      The benefits of multi-part upload are just too extensive and valuable.

      Now, an upload can be split into a maximum of 10,000 parts.

      And each part can range in size between 5 MB and 5 GB.

      The last part is left over, so it can be smaller than 5 MB if needed.

      Now, the reason why multi-part upload is so effective is that each individual part is treated as its own isolated upload.

      So each individual part can fail in isolation and be restarted in isolation, rather than needing to restart the whole thing.

      So this means that the risk of uploading large amounts of data to S3 is significantly reduced.

      But not only that, it means that because we're uploading lots of different individual bits of data, it improves the transfer rate.

      The transfer rate of the whole upload is the sum of all of the individual parts.

      So you get much better transfer rates by splitting this original blob of data into smaller individual parts and then uploading them in parallel.

      It means that if you do have any single stream limitations on your ISP or any network inefficiencies by uploading multiple different streams of data, then you more effectively use the internet bandwidth between you and the S3 endpoint.

      Now, next, I want to talk about a feature of S3 called Accelerated Transfer.

      To understand Accelerated Transfer, it's first required to understand how global transfer works to S3 buckets.

      Let's use an example.

      Let's say that the Animals for Life Organization has a European campaign which is running from the London office.

      For this campaign, there'll be data from staff in the field.

      And let's say that we have three teams dedicated to this campaign, one in Australia, one in South America, and one on the West Coast of the US.

      Now, the S3 bucket, which is being used by the campaign staff, has been created in the London region.

      So this is how this architecture locks.

      We've got three geographically spread teams who are going to be uploading data to an S3 bucket that's located within the UK.

      Now, it might feel like when you upload data to S3, your data would go in a relatively straight line, the most efficient line to its destination.

      Now, this is not how networking works.

      How networking works is that it is possible for the data to take a relatively indirect path.

      And the data can often slow down as it moves from hop to hop on the way to its destination.

      In some cases, the data might not be routed the way you expect.

      I've had data, for instance, routed from Australia to the UK, but taking the alternative path around the world.

      It's often not as efficient as you expect.

      Remember, S3 is a public service, and it's also regional.

      In the case of the Australian team, their data would have to transit across the public internet all the way from Australia to the UK before it enters the AWS public zone to communicate with S3.

      And we have no control over the public internet data path.

      Routers and ISPs are picking this path based on what they think is best and potentially commercially viable.

      And that doesn't always align with what offers the best performance.

      So using the public internet for data transit is never an optimal way to get data from source to destination.

      Luckily, as Solutions Architects, we have a solution to this, which is S3 transfer acceleration.

      Transfer acceleration uses the network of AWS edge locations, which are located in lots of convenient locations globally.

      An S3 bucket needs to be enabled for transfer acceleration.

      The default is that it's switched off, and there are some restrictions for enabling it.

      The bucket name cannot contain periods, and it needs to be DNS compatible in its naming.

      So keep in mind those two restrictions.

      But assuming that's the case, once enabled, data being uploaded by our field workers, instead of going back to the S3 bucket directly, it immediately enters the closest, best performing AWS edge location.

      Now this part does occur over the public internet, but geographically, it's really close, and it transits through fewer normal networks, so it performs really well.

      At this point, the edge locations transit the data being uploaded over the AWS global network, a network which is directly under the control of AWS, and this tends to be a direct link between these edge locations and other areas of the AWS global network, in this case, the S3 bucket.

      Remember, the internet is a global, multi-purpose network, so it has to have lots of connections to other networks, and many stops along the way, where traffic is routed from network to network, and this just slows performance down.

      Think of the internet as the normal public transit network, when you need to transit from bus to train to bus to bus, to get to a far-flung destination.

      The normal transit network, whilst it's not the highest performance, is incredibly flexible, because it allows you to get from almost anywhere to almost anywhere.

      The internet is very much like that.

      It's not designed primarily for speed.

      It's designed for flexibility and resilience.

      The AWS network, though, is purpose-built to link regions to other regions in the AWS network, and so this is much more like an express train, stopping at only the source and destination.

      It's much faster and with lower consistent latency.

      Now, the results of this, in this context, are more reliable and higher performing transfers between our field workers and the S3 bucket.

      The improvements can vary, but the benefits achieved by using transfer acceleration improve the larger the distance between the upload location and the location of the S3 bucket.

      So in this particular case, transferring data from Australia to a bucket located in Europe, you'll probably see some significant gains by using transfer acceleration.

      The worse the initial connection, the better the benefit by using transfer acceleration.

      Okay, so now it's time for a demonstration.

      In the next lesson, I just want to take a few moments to show you an example of how this works.

      I want to show you how to enable the feature on an S3 bucket, and then demonstrate some of the performance benefits that you can expect by using an AWS-provided tool.

      So go ahead, finish this video, and when you're ready, you can join me in the demo lesson.

    1. Welcome back and in this lesson I'm going to cover object versioning and MFA delete, two essential features of S3.

      These are two things I can almost guarantee will feature on the exam and almost every major project I can involved in has needed solid knowledge of both.

      So let's jump in and get started.

      Object versioning is something which is controlled at a bucket level.

      It starts off in a disabled state.

      You can optionally enable versioning on a disabled bucket, but once enabled you cannot disable it again.

      Just to be super clear, you can never switch bucket versioning back to disabled once it's been enabled.

      What you can do though is suspend it and if desired a suspended bucket can be re-enabled.

      It's really important for the exam to remember these stage changes.

      So make a point of noting them down and when revising try to repeat until it sticks.

      So a bucket starts off as disabled, it can be re-enabled again, an enabled bucket can be moved to suspended and then moved back to enabled.

      But the important one is that enabled bucket can never be switched back to disabled.

      That is critical to understand for the exam.

      So you can see many trick questions which will test your knowledge on that point.

      Without versioning enabled on a bucket, each object is identified solely by the object key, its name, which is unique inside the bucket.

      If you modify an object, the original version of that object is replaced.

      Versioning lets you store multiple versions of an object within a bucket.

      Any operations which would modify an object, generate a new version of that object and leave the original one in place.

      For example, let's say I have a bucket and inside the bucket is a picture of one of my cats, Winky.

      So the object is called Winky.JPEG.

      It's identified in the bucket by the key, essentially its name, and the key is unique.

      If I modify the Winky.JPEG object or delete it, those changes impact this object.

      Now there's an attribute of an object which I haven't introduced yet and that's the ID of the object.

      When versioning on a bucket is disabled, the ID of the object in that bucket are set to null.

      That's what versioning being off on a bucket means.

      All of the objects have an ID of null.

      Now if you upload or put a new object into a bucket with versioning enabled, then S3 allocates an ID to that object.

      In this case, 111, 111.

      If any modifications are made to this object, so let's say somebody accidentally overrides the Winky.JPEG object with the dog picture, but still calls it Winky.JPEG.

      S3 doesn't remove the original object.

      It allocates a new ID to the newer version and it retains the old version.

      The newest version of any object in a version-enabled bucket is known as the current version of that object.

      So in this case, the object called Winky.JPEG with an ID of 2222222, which is actually a dog picture, that is the current version of this object.

      Now if an object is accessed without explicitly indicating to S3 which version is required, then it's always the current version which will be returned.

      But you've always got the ability of requesting an object from S3 and providing the ID of a specific version to get that particular version back rather than the current version.

      So versions can be individually accessed by specifying the ID, and if you don't specify the ID, then it's assumed that you want to interact with the current version, the most recent version.

      Now versioning also impacts deletions.

      Let's say we've got these two different versions of Winky.JPEG stored in a version-enabled bucket.

      If we indicate to S3 that we want to delete the object and we don't give any specific version ID, then what S3 will do is try a new special version of that object known as a delete marker.

      Now the delete marker essentially is just a new version of that object, so S3 doesn't actually delete anything, but the delete marker makes it look deleted.

      In reality though, it's just hidden.

      The delete marker is a special version of an object which hides all previous versions of that object.

      But you can delete the delete marker which essentially undeletes the object, returning the current version to being active again, and all the previous versions of the object still exist, accessible using their unique version ID.

      Now even with versioning enabled, you can actually fully delete a version of an object, and that actually really deletes it.

      To do that, you just need to delete an object and specify the particular version ID that you want to remove.

      And if you are deleting a particular version of an object and the version that you're deleting is the most recent version, so the current version, then the next most recent version of that object then becomes the current version.

      Now some really important points that you need to be aware about object versioning.

      I've mentioned this at the start of the lesson, it cannot be switched off, it can only be suspended.

      Now why that matters is that when versioning is enabled on a bucket, all the versions of that object stay in that bucket, and so you're consuming space for all of the different versions of an object.

      If you have one single object that's 5 gig in size, and you have five versions of that object, then that's 5 times 5 gig of space that you're consuming for that one single object, and it's multiple versions.

      And logically, you'll build for all of those versions of all of those objects inside an S3 bucket, and the only way that you can zero those costs out is to delete the bucket and then re-upload all those objects to a bucket without versioning enabled.

      That's why it's important that you can't disable versioning.

      You can only suspend it, and when you suspend it, it doesn't actually remove any of those old versions, so you're still built for them.

      Now there's one other relevant feature of S3 which does make it to the exam all the time, and that's known as MFA delete.

      Now MFA delete is something that's enabled within the versioning configuration on a bucket.

      And when you enable MFA delete, it means that MFA is required to change bucket versioning state.

      So if you move from enable to suspend it or vice versa, you need this MFA to be able to do that, and also MFA is required to delete any versions of an object.

      So to fully delete any versions, you need this MFA token.

      Now the way that this works is that when you're performing API calls in order to change a bucket to versioning state or delete a particular version of an object, you need to provide the serial number of your MFA token as well as the code that it generates.

      You concatenate both of those together, and you pass that along with any API calls to interact how you delete versions or change the versioning state of a bucket.

      Okay, so that's all of the theory for object versioning inside S3.

      And at this point, that's everything I wanted to cover in this license.

      I'll go ahead and complete the video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back in this demo lesson you're going to gain some practical experience of working with the versioning feature of S3.

      So to get started just make sure that you're logged in to the management account of the organization, so the general account, and then make sure that you've got the Northern Virginia region selected, so US-EAS-1.

      Now there is a link attached to this lesson which you need to click on and then extract.

      This is going to contain all of the files that you'll be using throughout this demo.

      So go ahead and click on that link, extract it, and it should create a folder called S3_Versioning.

      Once you've confirmed that you're logged in and have the right region selected, then go ahead and move to the S3 console.

      So you can get there either using the recently visited services, or you can type S3 into the Find Services box and click to move to the S3 console.

      Now to demonstrate versioning we're going to go ahead and create an S3 bucket, we're going to set it up for static website hosting, enable versioning, and then experiment with some objects and just observe how versioning changes the default behavior inside an S3 bucket.

      So go ahead and click on Create bucket.

      As long as the bucket name is unique, its specific name isn't important because we won't be using it with Route 53.

      So just give the bucket a name and make sure that it's something unique.

      So I'm going to use AC_Bucket_13337.

      You should pick something different than me and different from something that any other student would use.

      Once you've selected a unique bucket name, just scroll down and uncheck Block All Public Access.

      We're going to be using this as a static website hosting bucket, so this is fine.

      And we'll need to acknowledge that we understand the changes that we're making, so check this box, scroll down a little bit more, and then under bucket versioning we're going to click to enable versioning.

      Keep scrolling down and at the bottom click on Create bucket.

      Next, go inside the bucket, click on Properties, scroll all the way down to the bottom, and we need to enable static website hosting.

      So click on Edit, check the box to enable static website hosting.

      For hosting type, we'll set it to host a static website, and then for the index document, just type index.html, and then for the error document, type error.html.

      Once you've set both of those, you can scroll down to the bottom and click on Save Changes.

      Now as you learned in the previous demo lesson, just enabling static website hosting isn't enough to allow access, we need to apply a bucket policy.

      So click on the permissions tab, scroll down, and under bucket policy click on Edit.

      Now inside the link attached to this lesson, which you should have downloaded and extracted, there should be a file called bucket_policy.json, which is an example bucket policy.

      So go ahead and open that file and copy the contents into your clipboard, move back to the console and paste it into the policy box, and we need to replace this example bucket placeholder with the ARN for this bucket.

      So copy the bucket ARN into your clipboard by clicking this icon.

      Because this ARN references objects in this bucket, and we know this because it's got forward slash star at the end, we need to replace only the first part of this placeholder ARN with the actual bucket ARN from the top.

      So select from the A all the way up to the T, so not including the forward slash and the star, and then paste in the bucket ARN that you copied onto your clipboard.

      Once you've done that, you can scroll down and then click on Save Changes.

      Next, click on the objects tab, and we're going to upload some of the files that you downloaded from the link attached to this lesson.

      So click on Upload, and first we're going to add the files.

      So click on Add Files, then you'll need to go to the location where you downloaded and extracted the file that's attached to this lesson.

      And once you're there, go into the folder called S3_Versioning, and you'll see a folder called Website.

      Open that folder, select index.html and click on Open, and then click on Add Folder, and select the IMG folder that's also in that same location.

      So select that folder and then click on Upload.

      So this is going to upload an index.html object, and it's going to upload a folder called IMG which contains winky.jpeg.

      Once you've done that, scroll down to the bottom and just click on Upload.

      Now once the upload's completed, you can go ahead and click on Close, and what you'll see in the Objects dialog inside the bucket is index.html and then a folder called IMG.

      And as we know by now, S3 doesn't actually have folders it uses prefixes, but if we go inside there, you'll see a single object called winky.jpeg.

      Now go back to the bucket, and what we're going to do is click on Properties, scroll down to the bottom, and then click on this icon to open our bucket in a new browser tab.

      All being well, you should see AnimalsForLife.org, Animal of the Week, and a picture of my one-eyed cat called winky.

      So this is using the same architecture as the previous demo lesson where you experienced static website hosting.

      What we're going to do now though is experiment with versions.

      So go back to the main S3 console, scroll to the top, and click on Objects.

      So because we've got versioning enabled on this bucket, as I talked about in the previous theory lesson, it means that every time you upload an object to this S3 bucket, it's assigned a version ID.

      And if you upload an object with the same name, then instead of overwriting that object, it just creates a new version of that object.

      Now with versioning enabled and using the default settings, we don't see all the individual versions, but we can elect to see them by toggling this Show Versions toggle.

      So go ahead and do that.

      Now you'll see that every object inside the S3 bucket, you'll see a particular version ID, and this is a unique code which represents this particular version of this particular object.

      So if we go inside the IMG folder, you'll see that we have the same for winkeep.jpeg.

      Toggle Show Versions to Disable, and you'll see that that version ID disappears.

      What I want you to do now is to click on the Upload button inside this IMG folder.

      So click on Upload, and then click on Add Files.

      Now inside this Lessons folder, so S3 versioning, at the top level you've got a number of folders.

      You have Website, which is what you uploaded to this S3 bucket, and this Image folder contains winkeep.jpeg.

      So this is a particular file, winkeep.jpeg, that contains the picture of winkeep my one-eyed cat.

      Now if you expand version 1 and version 2, you might be able to tell that version 1 is the same one-eyed cat, and we can expand that and say that it is actually winkeep.

      Inside version 2 we have an object with the same name, but if we expand this, this is not winkeep, this is a picture of truffles.

      So let's say that an administrator of this bucket makes a mistake and uploads this second version of winkeep.jpeg, which is not actually winkeep, it's actually truffles the cat.

      But let's say that we do this, so we select winkeep.jpeg from the version 2 folder, and we click on Open.

      Once we've selected that for upload, we scroll all the way down to the bottom and click on Upload.

      That might take a few seconds to complete the upload because these are relatively large image files, but once it's uploaded you can click on Close.

      So now we're still inside this image folder, and if we refresh, all we can see is one object, winkeep.jpeg.

      So it looks with this default configuration of the user interface, like we've overwritten a previous object with this new object.

      And if we go back to the tab which has got the static website open and hit refresh, you'll see that this image has indeed been replaced by the truffles image.

      So even though it's called winkeep.jpeg, this is clearly truffles.

      Now if we go back to the S3 console, and now if we enable the versions toggle, now we can see that we've got two different versions of this same object.

      We've got the original version at the bottom and a new version at the top.

      And note how both of these have different version IDs.

      Now what S3 does is it always picks the latest version whenever you use any operations which simply request that one object.

      So if we just request the object like we're doing with the static website hosting, then it will always pick the current or the latest version of this object.

      But we do still have access to the older versions because we have versioning enabled on this bucket.

      Nothing is ever truly deleted as long as we're operating with objects.

      So let's experiment with exactly what functionality this gives us.

      Go ahead and toggle show versions.

      Once you've done that, select the winkeep.jpeg object and then click delete.

      You'll need to type or copy and paste delete into this delete objects box and then click on delete.

      Before we do that, note what it says at the top.

      Deleting the specified objects adds delete markers to them.

      If you need to undo the delete action, you can delete the delete markers.

      So let's explore what this means.

      Go ahead and click on delete objects.

      And once it's completed, click on close.

      Now how this looks at the moment, we're still in the image folder.

      And because we've got show version set to off, it looks like we deleted the object.

      But this is not what's occurred because we've got versioning enabled.

      What's actually occurred is this is added a new version of this object.

      But instead of an actual new version of the object, it's simply added a delete marker as that new version.

      So if we toggle show versions back to on, now what we see are the previous versions of winkeep.jpeg.

      So the original version at the bottom and the one that we replaced in the middle.

      And then at the top we have this delete marker.

      Now the delete marker is the thing which makes it look to be deleted in the console UI when we have show version set to off.

      So this is how S3 handles deletions when versioning is enabled.

      If you're interacting with an object and you delete that object, it doesn't actually delete the object.

      It simply adds a delete marker as the most recent version of that object.

      Now if we just select that delete marker and then click on delete, that has the effect of undeleting the object.

      Now it's important to highlight that because we're dealing with object versions, anything that we do is permanent.

      If you're operating with an object and you have versioning enabled on a bucket, if you overwrite it or delete it, all it's going to do is either add a new version or add a delete marker.

      When you're operating with versions, everything is permanent.

      So in this case we're going to be permanently deleting the delete marker.

      So you need to confirm that by the typing or copying and pasting permanently delete into this box and click on delete objects.

      What this is going to do is delete the delete marker.

      So if we click on close, now we're left with these two versions of winkeep.jpeg so we've deleted the delete marker.

      If we toggle show versions to off, we can see that we now have our object back in the bucket.

      If we go back to static website hosting and refresh, we can see though that it's still truffle.

      So this is a mistake.

      It's not actually winky in this particular image.

      So what we can do is go back to the S3 console, we can enable show versions.

      We know that the most recent version is actually truffles rather than winky.

      So what we can do is select this incorrect version, so the most recent version and select delete.

      Now again, we're working with an object version.

      So this is permanent.

      You need to make sure that this is what you intend.

      In our case it is.

      So you need to either type or copy and paste permanently delete into the box and click on delete objects.

      Now this is going to delete the most recent version of this object.

      What happens when you do that is it makes the next most recent version of that object the current or latest version.

      So now this is the original version of winky.jpeg, the one that we first uploaded to this bucket.

      So this is now the only version of this object.

      If we go back to the static website hosting tab and hit refresh, this time it loads the correct version of this image.

      So this is actually winky my one-eyed cat.

      So this is how you can interact with versioning in an S3 bucket.

      Whenever it's enabled, it means that whenever you upload an object to the same name instead of overwriting, it simply creates a new version.

      Whenever you delete an object, it simply adds a delete marker.

      When you're operating with objects, it's always creating new versions or adding delete markers.

      But when you're working with particular versions rather than objects, any operations are permanent.

      So you can actually delete specific versions of an object permanently and you can delete delete markers to undelete that object.

      Now it's not possible to turn off versioning on a bucket.

      Once it's enabled on that bucket, you don't have the ability to disable it.

      You only have the ability to suspend it.

      Now when you suspend it, it stops new versions being created, but it does nothing about the existing versions.

      The only way to remove the additional costs for a version-enabled bucket is either to delete the bucket and then reload the objects to a new bucket, or go through the existing bucket and then manually purge any specific versions of objects which aren't required.

      So you need to be careful when you're enabling versioning on a bucket because it can cause additional costs.

      If you have a bucket where you're uploading objects over and over again, specifically of their large objects, then if you have versioning enabled, you can incur significantly higher costs than if you have a bucket which doesn't have a versioning enabled.

      So that's something you need to keep in mind.

      If you enable versioning, you need to manage those versions of those objects inside the bucket.

      With that being said, let's tidy up.

      So let's go back to the main S3 console, select the bucket, click on Empty, copy and paste or type "Permanently Delete" and click on Empty.

      When it's finished, click on Exit, and with the bucket still selected, click on Delete.

      Copy and paste or type the name of the bucket and confirm it with the delete bucket.

      I want you to build out the accounties back in the same state as it was at the start of this demo lesson.

      Now at this point, that's everything that I want you to do in this demo lesson.

      You've gained some practical exposure with how to deal with object versions inside an S3 bucket.

      At this point, go ahead and complete this video, and when you're ready, I'll afford you joining me in the next lesson.

    1. Welcome back.

      And in this demo lesson, you're going to get some experience using the S3 static website hosting feature, which I talked about in the previous lesson.

      Now, to get started, just make sure that you're logged in to the management account of the organization, and that you're using the IAM Admin user.

      So this just makes sure that you have admin permissions over the general or management account of the organization.

      Also, make sure that you have the Northern Virginia region selected, which is US-EAST-1.

      Normally with S3, when you're interacting with the product, you're doing so using the AWS console UI or the S3 APIs.

      And in this demo lesson, you'll be enabling a feature which allows S3 to essentially operate as a web server.

      It allows anybody with a web browser to interact with an S3 bucket, load an index page, and load pictures or other media that are contained within that bucket using standard HTTP.

      So that's what we're going to do.

      And to get started, we need to move across to the S3 console.

      So either use S3 in recently visited services, or you can click on the services dropdown, type S3, and then click it in this list.

      Now that we're at the S3 console, we're going to create an S3 bucket.

      Now, if you chose to register a domain earlier in the course, like I did, so I registered animalsforlife.io, then we're going to connect this S3 bucket with the custom domain that we registered so we can access it using that domain.

      If you chose not to use a domain, then don't worry, you can still do this demo.

      What you need to do is to go ahead and click on Create a bucket.

      Now for the bucket name, if you are not using a custom domain, then you can enter whatever you want in this bucket name as long as it's unique.

      If you did register a custom domain and you want to use this bucket with that domain, then you need to enter a DNS formatted bucket name.

      So in my case, I'm going to create a bucket which is called Top 10.

      It's going to store the world's best cappages, the Top 10 cappages in the world ever.

      And it's going to be part of the animalsforlife.io domain.

      And so at the end of this, I'm going to add dot and then animalsforlife.io.

      And if you've registered your own custom domain, then obviously you need to add your own domain at the end.

      You can't use the same name as me.

      Once you've entered that name and just scroll down and uncheck Block All Public Access, this is a safety feature of S3.

      But because we're intentionally creating an S3 bucket to be used as a static website, we need to uncheck this box.

      Now, unchecking this box means that you will be able to grant public access.

      It doesn't mean that public access is granted automatically when you uncheck this box.

      They're separate steps.

      You will, though, need to acknowledge that you understand the risks of unticking that box.

      So check this box just to confirm that you understand.

      We'll be carefully configuring the security so you don't have to worry about any of those risks.

      And once you've set that, we can leave everything else as default.

      So just scroll all the way down to the bottom and click on Create Bucket.

      So the bucket's been created, but right now, this only allows access using the S3 APIs or the console UI.

      So we need to enable static website hosting.

      Now, to do that, we're going to click on the bucket.

      Once we have the bucket open, we're going to select the Properties tab.

      On the Properties tab, scroll all the way down to the bottom.

      And right at the very bottom, we've got static website hosting.

      And you need to click on the Edit button next to that.

      It's a simple yes or no choice at this point.

      So check the box to enable static website hosting.

      There are a number of different types of hosting.

      You can either just host a static website, which is what we'll choose, or you can redirect requests for an object.

      So this allows you to redirect to a different S3 bucket.

      We'll be covering this later in the course for now.

      Just leave this selected.

      So host a static website.

      Now, in order to use the static website hosting feature, you'll need to provide S3 with two different documents.

      The index document is used as the home or default page for the static website hosting.

      So if you don't specify a particular object when you're browsing to the bucket, for example, winky.jpg, if you just browse to the bucket itself, then the index document is used.

      And we're going to specify index.html.

      So this means that an object called index.html will be loaded if we don't specify one.

      Now, the error document is used whenever you have any errors.

      So if you specify that you want to retrieve an object from the bucket, which doesn't exist, the error document is used.

      And for the error document, we're going to call this error.html.

      So these two values always need to be provided when you enable static website hosting.

      So now we've provided those, we can scroll down and click on save changes.

      Now that that feature's enabled, if we just scroll all the way down to the bottom, you'll see that we have a URL for this bucket.

      So go ahead and copy that into your clipboard.

      We're going to need this shortly.

      So this is the URL that you'll use by default to browse to this bucket.

      Now, next, what we need to do is to upload some objects to the bucket, which this static website hosting feature is going to use.

      Now, to do that, scroll all the way to the top and just click on objects and then click on upload.

      So this is the most recent UI version for S3.

      And so you have the ability to add files or add folders.

      Now, we're going to use both of these.

      We're going to use the add files button to add the index.html and the error.html.

      And we're going to use the add folder to add a folder of images.

      So first, let's do the add files.

      So click on add files.

      Now, attached to this video is a link which downloads all of the assets that you'll need for this demo.

      So go ahead and click on that link to download the zip file and then extract that zip file to your local machine.

      And you'll need to move to the folder that you extracted from the zip file.

      It should be called static_website_hosting.

      So go to that folder.

      And then again, there should be a folder in there called website_files.

      So go ahead and click on there to go into that folder.

      Now, there are three things inside this folder, index.html, error.html and img.

      So we'll start by uploading both of these HTML documents.

      So select index.html and error.html and then click on open.

      And that will add both of these to this upload table.

      Next, click on add folder and then select the img folder and click on upload.

      So this has prepared all of these different objects ready to upload to this S3 bucket.

      If we scroll down, we'll see that the destination for these uploads is our S3 bucket and your name here will be different as long as it's the same as the name you picked for the bucket, that's fine.

      Go all the way to the bottom and then go ahead and click on upload.

      And that will upload the index.html, the error.html and then the folder called img as well as the contents of that folder.

      So at this point, that's all of the objects uploaded to the S3 bucket and we can go ahead and click on close.

      So now let's try browsing to this bucket using static website hosting.

      So go ahead and click on properties, scroll all the way down to the bottom and here we've got the URL for this S3 bucket.

      So go ahead and copy this into your clipboard, open a new tab and then open this URL or click on this symbol to open it in a new tab.

      What you'll see is a 403 forbidden error and this is an access denied.

      You're getting this error because you don't have any permissions to access the objects within this S3 bucket.

      Remember, S3 is private by default and just because we've enabled static website hosting doesn't mean that we have any permissions to access the objects within this S3 bucket.

      We're accessing this bucket as an anonymous or unauthenticated user.

      So we have no method of providing any credentials to S3 when we're accessing objects via static website hosting.

      So we need to give permissions to any unauthenticated or anonymous users to access the objects within this bucket.

      So that's the next thing we need to do.

      We need to grant permissions to be able to read these objects to any unauthenticated user.

      So how do we do that?

      The third method is to use a bucket policy.

      So that's what I'm gonna demonstrate in order to grant access to these objects.

      Now to add a bucket policy, we need to select the permissions tab.

      So click on permissions and then below block public access, there's a box to specify a bucket policy.

      So click on edit and we need to add a bucket policy.

      Now also in the folder that you extracted from this lessons zip file is a file called bucket_policy.json and this is a generic bucket policy.

      So this bucket policy has an effect of allow and it applies to any principle because we have this star wild card and because the effect is allow, it grants any principle, the ability to use the S3 get object action which allows anyone to read an object inside an S3 bucket and it applies to this resource.

      So this is a generic template, we need to update it, but go ahead and copy it into your clipboard, go back to the S3 console and paste it into this box.

      Now we need to replace this generic ARN, so this example bucket ARN.

      So what I want you to do is to copy this bucket ARN at the top of the screen.

      So copy this into your clipboard and we need to replace part of this template ARN with what we've just copied.

      Now an important point to highlight is that this ARN has forward slash star on the end because this ARN refers to any objects within this S3 bucket.

      So we need to select only the part before the forward slash.

      So starting at the A and then ending at the end of example bucket and then just paste in the ARN at our bucket that we just copied into our clipboard.

      What you should end up with is this full ARN with the name of the bucket that you created and then forward slash star.

      And once you've got that, go ahead and click on save changes.

      This applies a bucket policy which allows any principle, so even unauthenticated principles, the ability to get any of the objects inside this bucket.

      So this means that any principle will be able to read objects inside this bucket.

      At this point, assuming everything's okay, if you've still got the tab open to the bucket, then go back to that tab and hit refresh.

      And what you should see is the top 10 animals in the world.

      So position number one, we've got Merlin.

      At position number two, we've got Merlin again.

      Position number three, another Merlin.

      Four, still Merlin.

      And then Merlin again at number five.

      At number six, we've got Boris.

      So the token non Merlin cat.

      Number seven, Samson, another token non Merlin cat.

      And then number eight, we've got different cat one.

      He looks quite a lot like Merlin.

      Number nine, different cat two, again, kind of looks like Merlin.

      And then number 10, we've got the family.

      And then you might not have guessed this, but this entire top 10 contest was judged by, you guessed it, Merlin.

      So what you're loading here is the index.html document inside the bucket.

      So we haven't specified an object to load.

      And because of that, it's using the index document that we specified on the bucket.

      We can load the same object by typing specifically index.html on the end, and that will load in the same object.

      Now, if we specify an object which doesn't exist, so let's say we used wrong index.html, then instead of the index document, now it's going to load the error document.

      So this is the error document that you specified, which is loading error.html.

      So this is just an example of how you can configure an S3 bucket to act as a standard static website.

      So what it's doing is loading in the index.html object inside the bucket.

      And that index.html is loading in images, which are also stored in the bucket.

      So if I right click and copy the image location and open this in a new tab, this is essentially just loading this image from the same S3 bucket.

      So it's loading it from this folder called img, and it's called Merlin.jpeg.

      It's just an object loading from within the bucket.

      Now if I go back to the S3 console and just move across to the properties tab and then scroll down, so far in this lesson, you've been accessing this bucket using the bucket website endpoint.

      So this is an endpoint that's derived from the name of the bucket.

      Now your URL will be different because you will have called your bucket name something else.

      Now if you chose to register a custom domain name at the start of this course, you can customize this further.

      As long as you call the bucket the same as the DNS name that you want to use, you can actually use Route 53 to assign a custom DNS name for this bucket.

      So this part of the demo you'll only be able to do if you've registered a domain within Route 53.

      If you haven't, you can skip to the end of this demo where we're going to tidy up.

      But if you want to customize this using Route 53, then you can click on the services dropdown and type Route 53 and then click to move to the Route 53 console.

      Once you're there, you can click on hosted zones and you should have a hosted zone that matches the domain that you registered at the start of the course.

      Go inside that and click on create record.

      Now we're going to be creating a simple routing record.

      So make sure that's selected and then click on next.

      And we're going to define a simple record.

      Now I'm going to type the first part of the name of the bucket.

      So I used top10.animalsforlive.io is my bucket name.

      So I'm going to put top 10 in this box.

      Now, because we want to point this at our S3 bucket, we need to choose an endpoint in this dropdown.

      So click in this dropdown and then scroll down and we're going to pick alias to S3 website endpoint.

      So select that.

      Next, you need to choose the region and you should have created the S3 bucket in the US East 1 region because this is the default for everything that we do in the course.

      So go ahead and type US-EAS-1 and then select US East Northern Virginia and you should be able to click in enter S3 endpoint and select your bucket name.

      Now, if you don't see your bucket here, then either you've picked the wrong region or you've not used the same name in this part of the record name as you picked for your bucket.

      So make sure this entire name, so this component plus the domain that you use matches the name that you've selected for the bucket.

      Assuming it does, you should be able to pick your bucket in this dropdown.

      Once you've selected it, go ahead and click on define simple record.

      And once that's populated in the box, click on create records.

      Now, once this record's created, you might have to wait a few moments, but you should find that you can then open this bucket using this full DNS name.

      So there we go.

      It opens up the same bucket.

      So we've used Route 53 and we've integrated it using an alias to our S3 endpoint.

      Now, again, you can only do this if you create a bucket with the same name as the fully qualified domain name that we just configured.

      So this is an example of a fully qualified domain name.

      Now, this is the host component of DNS and this is the domain component.

      So together they make up a fully qualified domain name and for this to work, you need to create an S3 bucket with the same bucket name as this fully qualified domain name.

      And that's what I did at the start of this lesson, which is why it works for me.

      And as long as you've done the same, as long as you've registered a custom domain, as long as you've called the bucket the same as what you're creating within Route 53, then you should be able to reference that bucket and then access it using this custom URL.

      At this point, we're going to tidy up.

      So go back to the Route 53 console and select this record that you've created and then click on delete.

      You'll need to confirm it by clicking delete again.

      Then we need to go back to the S3 console, select the bucket that you've created, click on empty, and you'll need to either type or copy and paste, permanently delete into this box, and then click on empty.

      It'll take a few minutes to empty the bucket.

      Once it's completed, click on exit.

      And with the bucket still selected, click on delete to delete the bucket.

      And you'll need to confirm that by either typing or copy and pasting the name of the bucket and then click delete bucket.

      Now, at this point, that's everything that you need to do in this lesson.

      It's just an opportunity to experience the theory that you learned in the previous lesson.

      Now, there's a lot more that you can do with static website hosting and I'll be going into many more complex examples later on in the course.

      But for now, this is everything that you need to do.

      So go ahead and complete this video.

      And when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back.

      In this lesson, I want to talk about a feature of S3 which I use all the time, personally and when I'm doing consulting work for clients. That is S3 static website hosting.

      Until this point, we've been accessing S3 via the normal method which is using the AWS APIs. You might not have realized that, but that's how the AWS CLI tools and the console UI work behind the scenes. For instance, to access any objects within S3, we're using the S3 APIs, and assuming we're authenticated and authorized, we use the get object API call to access those resources.

      Now, accessing S3 using APIs is useful in certain situations because it's secure and flexible. But using static website hosting can make S3 infinitely more useful because it allows access via standard HTTP, like individuals using a web browser. So, you can use it to host almost anything, for example, a simple blog.

      Using static website hosting is pretty simple. You enable it, and in doing so, you have to set an index document and an error document. When you're using a website, if you access a particular page, say, for example, cats_are_amazing.html, then you will get access specifically to that page. If you don't specify a page, for example, netflix.com, you get what's called an index page, which is a default page returned to you when you aren't asking for anything specific. This is the entry point to most websites.

      So, when enabling static website hosting on an S3 bucket, we have to point the index document at a specific object in the S3 bucket. The error document is the same, but it's used when something goes wrong. So, if you access a file which isn't there or there is another type of server-side error, that's when the error document is shown.

      Now, both of these need to be HTML documents because the static website hosting feature delivers HTML files. When you enable this feature on a bucket, AWS creates a static website hosting endpoint, and this is a specific address that the bucket can be accessed from using HTTP. The exact name of this endpoint is influenced by the bucket name that you choose and the region that the bucket is in. You don't get to select this name; it's automatically generated by those two things.

      Now, you can use your own custom domain for a bucket, but if you do want to do that, then your bucket name matters. You can only use a custom domain with a bucket if the name of the bucket matches the domain. So, if I wanted to have a website called top10.animalsforlife.org, then my bucket name would need to be called top10.animalsforlife.org, and that's why I mentioned at the start of the course to get into the habit of reserving your website names by creating S3 buckets using those names.

      Static website hosting is great for things like hosting static websites such as blogs, but it's also good for other things. Let's take a look at two common examples. There are two specific scenarios which are perfect for S3: Offloading and out-of-band pages.

      With offloading, let's say you have a website which has a top 10 leaderboard of all of the best animals, and it runs on a computer service. Let's assume this is easy too, for now. The computer service does a few things: it delivers a dynamic HTML page and it delivers static media, in this example, images. That dynamic HTML page might need access to a database, so that's not suitable for static S3 hosting, but the static media that's sitting there waiting to be delivered, and in most cases, it probably makes up over 95% of the data volume that the computer service is delivering and likely almost all of the storage space.

      Computer services tend to be relatively expensive, so we can offload a lot of this to S3. What we can do is we can take all of the images, so all of the media that the computer service hosts, and we can move that media to an S3 bucket that uses static website hosting, so one which has static website hosting enabled. Then, when the computer service generates the HTML file and delivers this to the customer's browser, this HTML file points at the media that's hosted on the S3 bucket, so the media is retrieved from S3, not the computer service. S3 is likely to be much cheaper for the storage and delivery of any media versus computer service. S3 is custom-designed for the storage of large data at scale, and so generally, whenever you've got an architecture such as this, you should always consider offloading any large data to S3.

      Now, the other benefit that I wanted to specifically highlight is out-of-band pages. Now, out-of-band is an old telecommunications term. In an IT context, it generally means a method of accessing something that is outside of the main way. So, for example, you might use out-of-band server management, and this lets you connect to a management card that's in a server using the cellular network. That way, if the server is having networking issues with the normal access methods of the normal network, then you can still access it.

      In the context of this example, top10.animalsforlife.org might refer to an error or a status notification system. For example, with this example, the favorite animals page for Animals for Life. If that service was hosted on a computer service such as EC2, and we wanted a maintenance page to show during scheduled or unscheduled maintenance periods, it wouldn't make much sense to have this on the same server because if the server is being worked on, then it's inherently offline. Additionally, putting it on a different EC2 instance is also risky because if EC2 itself has issues, it might not let us show a status page.

      So what we do is we use out-of-band pages in this context and another service. If the server was offline for maintenance or it was experiencing stability or performance bugs, then we could change our DNS and point customers at a backup static website hosted on S3, and this could provide a status message or maybe the details for our business's support team.

      Now, the pricing structure for S3, once you understand it, is very simple, but it's formed of a number of major components. First, we've got the cost to store data on S3, and this is generally expressed as a per-gigabyte-month fee. So to store a gigabyte of data on S3 for one month, there's a certain cost, and if you store data for less than one month, then you only pay that component, and if you store less than one gig, you only pay that component, so it's a per-gig-month charge.

      Now, there's also a data transfer fee, so for every gigabyte of data that you transfer in and out of S3, there's a cost. Now, to transfer data into S3 is always free, so you're never charged for transferring data into S3. To transfer data out of S3, there is a per-gigabyte charge. Now, for storage and data transfer, they are both incredibly cheap. It's sort of the cheapest storage that's available, especially if you're storing large amounts of data, but there is also a third component that you do need to be aware of, and this is especially important when you're using static website hosting, and this is that you're charged a certain amount for requesting data. So, every time you perform an operation, every time you get, every time you list, every time you put, that's classed as an operation, and different operations in S3 have different costs per 1,000 operations.

      Now, the reason I mention this is if you're using static website hosting, you're generally not going to store a lot of data. You're also generally not going to transfer a lot of data because what's stored in the S3 bucket is likely to be very small, but if you have a large customer base and if this out-of-band website or this offloading bucket is actually being used heavily by your system, then you could be using a lot of requests, and so you need to be aware of the request charge per S3.

      Now, in terms of what's provided in the free tier, you're given 5 GB of monthly storage inside S3. You're allowed 20,000 GET requests and 2,000 PUT requests, so that will cover us for the demo lesson that we're going to do inside this course and probably most of the other activities that we'll do throughout the course, but if you're going to use S3 for any real-world usage, then you will be billed for that usage.

      Now, I run a personal blog, control.io, and that runs from S3 using the static website hosting feature, and because I post certification articles, it does get some fairly heavy use. Now, in the entire time that I've run my personal blog, I think the most that I've ever been charged for S3 usage is 17 cents in one month. So when I talk about being charged for S3, I'm going to mention it whenever we go beyond this free tier, but keep in mind that often I'm talking about really tiny amounts of money relative to the value that you're getting. So, you can store a lot in S3 and use it to deliver a lot of data and often be charged a really tiny amount of money, often something that isn't noticeable on the bill of a production aid of the US account.

      Okay, that's enough on the theory of static website hosting, so now it's time for a demo, and in this demo, we're going to be using S3 to create a simple static website. Now, I think this demo is going to be a useful one because it brings together a few of the theory concepts that I've been talking about over the last few lessons. So, go ahead, mark this video as complete, and when you're ready, you can join me in the next demo lesson.

    1. Welcome back and in this lesson I want to start talking about S3 security in more detail. Starting with bucket policies which are a type of AWS resource policy. So by now you know the drill, let's jump in and get started.

      Now before we start I want to repeat one thing and you have heard me say this before, but I'm going to say it again over and over. S3 is private by default. Everything that we can do to control S3 permissions is based on this starting point. The only identity which has any initial access to an S3 bucket is the account root user of the account which owns that bucket, so the account which created it. Anything else, so any other permissions have to be explicitly granted. And there are a few ways that this can be done.

      The first way is using an S3 bucket policy. And an S3 bucket policy is a type of resource policy. A resource policy is just like an identity policy, but as the name suggests, they're attached to resources instead of identities, in this case an S3 bucket. Resource policies provide a resource perspective on permissions. The difference between resource policies and identity policies is all about this perspective. With identity policies you're controlling what that identity can access. With resource policies you're controlling who can access that resource. So it's from an inverse perspective. One is identities and one is resources.

      Now identity policies have one pretty significant limitation. You can only attach identity policies to identities in your own account. And so identity policies can only control security inside your account. With identity policies you have no way of giving an identity in another account access to an S3 bucket. That would require an action inside that other account. Resource policies allow this. They can allow access from the same account or different accounts because the policy is attached to the resource and it can reference any other identities inside that policy. So by attaching the policy to the resource and then having flexibility to be able to reference any other identity, whether they're in the same account or different accounts, resource policies therefore provide a great way of controlling access for a particular resource, no matter what the source of that access is.

      Now think about that for a minute because that's a major benefit of resource policies, the ability to grant other accounts access to resources inside your account. They also have another benefit, resource policies can allow or deny anonymous principals. Identity policies by design have to be attached to a valid identity in AWS. You can't have one attached to nothing. Resource policies can be used to open a bucket to the world by referencing all principals, even those not authenticated by AWS. So that's anonymous principals. So bucket policies can be used to grant anonymous access.

      So two of the very common uses for bucket policies are to grant access to other AWS accounts and anonymous access to a bucket. Let's take a look at a simple visual example of a bucket policy because I think it will help you understand how everything fits together. There's a demo lesson coming up soon where you'll implement one as part of the mini project. So you will get some experience soon enough of how to use bucket policies.

      Let's say that we have an AWS account and inside this account is a bucket called Secret Cat Project. Now I can't say what's inside this bucket because it's a secret, but I'm sure that you can guess. Now attached to this bucket is a bucket policy. Resource policies have one major difference to identity policies and that's the presence of an explicit principal component. The principal part of a resource policy defines which principals are affected by the policy. So the policy is attached to a bucket in this case, but we need a way to say who is impacted by the configuration of that policy. Because a bucket policy can contain multiple statements, there might be one statement which affects your account and one which affects another account, as well as one which affects a specific user, the principal part of a policy or more specifically the principal part of a statement in a policy defines who that statement applies to, which identity is which principals.

      Now in an identity policy this generally isn't there because it's implied that the identity which the policy is applied to is the principal. That's logical right? Your identity policy by definition applies to you so you are the principal. So a good way of identifying if a policy is a resource policy or an identity policy is the presence of this principal component. If it's there it's probably a resource policy. In this case the principal is a wild card, a star, which means any principal. So this policy applies to anyone accessing the S3 bucket.

      So let's interpret this policy. Well first the effect is allow and the principal is star, so any principal. So this effect allows any principal to perform the action S3 get object on any object inside the secret cat project S3 bucket. So in effect it allows anyone to read any objects inside this bucket. So this would equally apply to identities in the same AWS account as the bucket. It could also apply to other AWS accounts, or partner account. And crucially it also applies to anonymous principals. So principals who haven't authenticated to AWS. Bucket policies should be your default thought when it comes to granting anonymous access to objects in buckets and they're one way of granting external accounts that same access. They can also be used to set the default permissions on a bucket. If you want to grant everyone access to Boris's picture for example and then grant certain identities extra rights or even deny certain rights then you can do that. Bucket policies are really flexible. They can do many other things.

      So let's quickly just look at a couple of common examples. Bucket policies can be used to control who can access objects even allowing conditions which block specific IP addresses. In this example this bucket policy denies access to any objects in the secret cat project bucket unless your IP address is 1.3.3.7. The condition block here means this statement only applies if this condition is true. So if your IP address, the source IP address is not 1.3.3.7, then the statement applies and access is denied. If your IP address is 1.3.3.7, then this condition is not met because it's a not IP address condition. So if your IP address is this IP address, the condition is not matched and you get any other access that's applicable. Essentially this statement, which is a deny, does not apply.

      Now, bucket policies can be much more complex. In this example, one specific prefix in the bucket, remember this is what a folder really is inside a bucket, so one specific prefix called Boris is protected with MFA. It means that accesses to the Boris folder in the bucket are denied if the identity that you're using does not use MFA. The second statement allows read access to objects in the whole bucket. Because an explicit deny overrides an allow, the top statement applies to just that specific prefix in the bucket, so just Boris. Now, I won't labour on about bucket policies because we'll be using them a fair bit throughout the course, but they can range from simple to complex. I will include a link in the lesson description with some additional examples that you can take a look through if you're interested.

      In summary though, a resource policy is associated with a resource. A bucket policy, which is a type of resource policy, is logically associated with a bucket, which is a type of resource. Now, there can only be one bucket policy on a bucket, but it can have multiple statements. If an identity inside one AWS account is accessing a bucket, also in that same account, then the effective access is a combination of all of the applicable identity policies plus the resource policy, so the bucket policy. For any anonymous access, so access by an anonymous principal, then only the bucket policy applies, because logically, if it's an anonymous principal, it's not authenticated and so no identity policies apply.

      Now, if an identity in an external AWS account attempts to access a bucket in your account, your bucket policy applies as well as anything that's in their identity policies. So there's a two-step process if you're doing cross-account access. The identity in their account needs to be able to access S3 in general and your bucket, and then your bucket policy needs to allow access from that identity, so from that external account.

      Now, there is another form of S3 security. It's used less often these days, but I wanted to cover it anyway. Access control lists or ACLs are ways to apply security to objects or buckets. There is a sub-resource of that object or of that bucket. Remember in the S3 introduction lesson earlier in the course, I talked about sub-resources. Well, this is one of those sub-resources. Now, I almost didn't want to talk about ACLs because they are legacy. AWS don't even recommend their use and prefer that you use bucket policies or identity policies. But as a bare minimum, I want you to be aware of their existence.

      Now, part of the reason that they aren't used all that often and that bucket policies have replaced much of what they do is that they're actually inflexible and only allow very simple permissions. They can't have conditions like bucket policies and so you're restricted to some very broad conditions. Let me show you what I mean. This is an example of what permissions can be controlled using an ACL. Now, apologies for the wall of text, but I think it's useful to visualize it all at once. There are five permissions which can be granted in an ACL. Read, write, readACP, writeACP and full control. That's it. So it's already significantly less flexible than an identity or a resource policy. What these five things do depend on if they're applied to a bucket or an object. Read permissions, for example, on a bucket allow you to list all objects in that bucket, whereas write permissions on a bucket allow the grantee, which is the principal being granted those permissions, the ability to overwrite and delete any object in that bucket. Read permissions on an object allow the grantee just to read the object specifically as well as its metadata.

      Now, with ACLs you either configure an ACL on the bucket, or you configure the ACL on an object. But you don't have the flexibility of being able to have a single ACL that affects a group of objects. You can't do that. That's one of the reasons that a bucket policy is significantly more flexible. It is honestly so much less flexible than a bucket policy to the extent where I won't waste your time with it anymore. It's legacy, and I suspect at some point it won't be used anymore. If there are any specific places in the course which do require knowledge of ACLs, I'll mention it. Otherwise, it's best to almost ignore the fact that they exist.

      Now, before we finish up, one final feature of S3 permissions, and that's the block public access settings. In the overall lifetime of the S3 product, this was actually added fairly recently, and it was added in response to lots of public PR disasters where buckets were being configured incorrectly and being set so that they were open to the world. This resulted in a lot of data leaks, and the root cause was a mixture of genuine mistakes or administrators who didn't fully understand the S3 permissions model.

      So consider this example, an S3 bucket with resource permissions granting public access. Until block public access was introduced, if you had public access configured, the public could logically access a bucket. Public access in this sense is read only to any objects defined in a resource policy on a bucket, so there's no restrictions. Public access is public access. Block public access added a further level of security, another boundary. And on this boundary is the block public access settings, which apply no matter what the bucket policies say, but they apply to just the public access, so not any of the defined AWS identities. So these settings will only apply to an anonymous principal, somebody who isn't an AWS identity, attempt to access a bucket using these public access configurations.

      Now these settings can be set when you create the bucket and adjusted afterwards. They're pretty simple to understand. You can choose the top option which blocks any public access to the bucket, no matter what the resource policy says. It's a full override, a failsafe. Or you can choose the second option which allows any public access granted by any existing ACLs when you enable the setting but it blocks any new ones. The third option blocks any public access granted by ACLs no matter if it was enabled before or after the block public access settings were enabled. The fourth setting allows any existing public access granted by bucket policies or access point policies so anything enabled at the time when you enable this specific block public access setting, they're allowed to continue but it blocks any new ones. The fifth option blocks both existing and new bucket policies from granting any public access.

      Now they're simple enough and they function as a final failsafe. If you're ever in a situation where you've granted some public access and it doesn't work, these are probably the settings which are causing that inconsistency. And don't worry, I'll show you where these are accessed in the demo lesson.

      Now before we finish up, just one final thing I want to cover and this is an exam at PowerUp. So these are just some key points on how to remember all of the theory that I've discussed in this lesson. When I first started in AWS, I found it hard to know from instinct when to use identity policies versus resource policies versus ACLs. Choosing between resource policies and identity policies much of the time is a preference thing. So do you want to control permissions from the perspective of a bucket or do you want to grant or deny access from the perspective of the identities accessing a bucket? Are you looking to configure one user accessing 10 different buckets or 100 users accessing the same bucket? It's often a personal choice. A choice on what makes sense for your situation and business. So there's often no right answer but there are some situations where one makes sense over the other.

      If you're granting or denying permissions on lots of different resources across an AWS account, then you need to use identity policies because not every service supports resource policies. And besides, you would need a resource policy for each service so that doesn't make sense if you're controlling lots of different resources. If you have a preference for managing permissions all in one place, that single place needs to be IAM, so identity policies would make sense. IAM is the only single place in AWS you can control permissions for everything. You can sometimes use resource policies but you can use IAM policies all the time. If you're only working with permissions within the same account so no external access, then identity policies within IAM are fine because with IAM you can only manage permissions for identities that you control in your account. So there are a wide range of situations where IAM makes sense and that's why most permissions control is done within IAM. But there are some situations which are different. You can use bucket policies or resource policies in general if you're managing permissions on a specific product. So in this case S3. If you want to grant a single permission to everybody accessing one resource or everybody in one account, then it's much more efficient to use resource policies to control that base level permission. If you want to directly allow anonymous identities or external identities from other AWS accounts to access a resource, then you should use resource policies.

      Now finally, and I know this might seem like I'm anti-access control list, which is true, but so are AWS, never use ACLs unless you really need to. And even then, consider if you can use something else. At this point in time, if you are using an ACL, you have to be pretty certain that you can't use anything else because they're legacy and their inflexible and AWS are actively recommending against their use. So keep that in mind.

      Okay, well that's all of the theory that I wanted to cover in this lesson. I know it's been a lot, but we do have to cover this detailed level of security because it's needed in the exam. And you'll be using it constantly throughout the rest of this section and the wider course. At this point, though, go ahead and complete this video. And when you're ready, you can join me in the next where I'm going to be talking about another exciting feature of S3.

    1. Welcome to this lesson, where I'm going to very briefly talk about a special type of IAM role, and that's service-linked roles.

      Now, luckily there isn't a great deal of difference between service-linked roles and IAM roles. They're just used in a very specific set of situations. So let's jump in and get started.

      So simply put, a service-linked role is an IAM role linked to a specific AWS service. They provide a set of permissions which is predefined by a service. These permissions allow a single AWS service to interact with other AWS services on your behalf.

      Now, service-linked roles might be created by the service itself, or the service might allow you to create the role during the setup process of that service. Service-linked roles might also get created within IAM.

      The key difference between service-linked roles and normal roles is that you can't delete a service-linked role until it's no longer required. This means it must no longer be used within that AWS service. So that's the one key difference.

      In terms of permissions needed to create a service-linked role, here's an example of a policy that allows you to create a service-linked role.

      You'll notice a few key elements in this policy. The top statement is an allow statement. The action is iam:CreateServiceLinkedRole. For the resource, it has SERVICE-NAME.amazonaws.com.

      The important thing here is not to try to guess this, as different services express this in different ways. The formatting can differ, and it's case-sensitive. I've included a link with an overview of these details attached to this lesson.

      When creating this type of policy to allow someone to create service-linked roles, you have to be careful to ensure you do not guess this element of a statement.

      Another important consideration with service-linked roles is role separation. When I talk about role separation, I'm not using it in a technical sense, but in a job role sense.

      Role separation is where you might give one group of people the ability to create roles and another group the ability to use them. For instance, we might want to give Bob, one of our users, the ability to use a service-linked role with an AWS service.

      This involves using the architecture of being able to take a service-linked role and assign it to a service. If you want to give Bob the ability to use a preexisting role with a service but not create or edit that role, you would need to provide Bob with PassRole permissions. This allows Bob to pass an existing role into an AWS service. It's an example of role separation, meaning Bob could configure a service with a role that has already been created by a member of the security team. Bob would just need ListRole and PassRole permissions on that specific role.

      This is similar to when you use a pre-created role, for example, with a CloudFormation stack. By default, when creating a CloudFormation stack, CloudFormation uses the permissions of your identity to interact with AWS. This means you need permissions not only to create a stack but also to create the resources that the stack creates. However, you can give, for example, a user like Bob the ability to pass a role into CloudFormation. That role could have permissions that exceed those which Bob directly has. So a role that Bob uses could have the ability to create AWS resources that Bob does not. Bob might have access to create a stack and pass in a role, but the role provides CloudFormation with the permissions needed to interact with AWS.

      PassRole is a method inside AWS that allows you to implement role separation, and it's something you can also use with service-linked roles. This is something I wanted to reiterate to emphasize that passing a role is a very important AWS security architecture.

      That is everything I wanted to cover in this very brief lesson. It's really just an extension of what you've already learned about IAM roles, and it's something you'll use in demo lessons elsewhere in the course.

      For now, I just want you to be aware of how service-linked roles differ from normal roles and how the PassRole architecture works. With that being said, that's everything I wanted to cover in this video.

      So go ahead and complete the video, and when you're ready, I look forward to you joining me in the next.

    1. Welcome back, and in this video, I want to talk about AWS Control Tower. This is a product which is becoming required knowledge if you need to use AWS in the real world. And because of this, it's starting to feature more and more in all of the AWS exams. I want this to be a lesson applicable to all of the AWS study paths, so think of this as a foundational lesson. And if required, for the course that you're studying, I might be going into additional detail. We do have a lot to cover, so let's jump in and get started.

      At a high level, Control Tower has a simple but wide-ranging job, and that's to allow the quick and easy setup of multi-account environments. You might be asking, "Doesn't AWS Organizations already do that?" Well, kind of. Control Tower actually orchestrates other AWS services to provide the functionality that it does, and one of those services is AWS Organizations. But it goes beyond that. Control Tower uses Organizations, IAM Identity Center, which is the product formerly known as AWS SSO. It also uses CloudFormation, AWS Config, and much more. You can think of Control Tower as another evolution of AWS Organizations adding significantly more features, intelligence, and automation.

      There are a few different parts of Control Tower which you need to understand, and it's worth really focusing on understanding the distinction now because we're going to be building on this later. First, we've got the Landing Zone, and simply put, this is the multi-account environment part of Control Tower. This is what most people will be interacting with when they think of Control Tower. Think of this like AWS Organizations only with superpowers. It provides, via other AWS services, single sign-on and ID Federation so you can use a single login across all of your AWS accounts, and even share this with your existing corporate identity store. And this is provided using the IAM Identity Center, again, the service formerly known as AWS SSO. It also provides centralized logging and auditing, and this uses a combination of CloudWatch, CloudTrail, AWS Config, and SNS. Everything else in the Control Tower product surrounds this Landing Zone, and I'll show you how this looks later in this lesson.

      Control Tower also provides guardrails, again, more detail on this is coming up soon. But these are designed to either detect or mandate rules and standards across all AWS accounts within the Landing Zone. You also have the Account Factory, which provides really cool automation for account creation, and adds features to standardize the creation of those accounts. This goes well beyond what AWS Organizations can do on its own, and I'll show you how this works over the rest of this lesson. And if applicable, for the path that you're studying, there will be a demo coming up elsewhere in the course. Finally, there's a dashboard which offers a single-page oversight of the entire organization. At a high level, that's what you get with Control Tower.

      Now, things always make more sense visually, so let's step through this high-level architecture visually, and I hope this will add a little bit more context. We start with Control Tower itself, which like AWS Organizations, is something you create from within an AWS account. And this account becomes the management account at the Landing Zone. At this top, most level within the management account, we have Control Tower itself, which orchestrates everything. We have AWS Organizations, and as you've already experienced, this provides the multi-account structure, so organizational units and service control policies. And then, we have single sign-on provided by the IAM Identity Center, which historically was known as AWS SSO. This allows for, as the name suggests, single sign-on, which means we can use the same set of internal or federated identities to access everything in the Landing Zone that we have permissions to. This works in much the same way as AWS SSO worked, but it's all set up and orchestrated by Control Tower.

      When Control Tower is first set up, it generally creates two organizational units. The foundational organizational units, which by default is called Security, and a custom organizational unit, which by default is named Sandbox. Inside the foundational or security organizational unit, Control Tower creates two AWS accounts, the Audit account and the Log Archive account. The Log Archive account is for users that need access to all logging information for all of your enrolled accounts within the Landing Zone. Examples of things used within this account are AWS Config and CloudTrail logs, so they're stored within this account so that they're isolated. You have to explicitly grant access to this account, and it offers a secure, read-only Archive account for logging.

      The Audit account is for your users who need access to the audit information made available by Control Tower. You can also use this account as a location for any third-party tools to perform auditing of your environment. It's in this account that you might use SNS for notifications of changes to governance and security policies, and CloudWatch for monitoring Landing Zone wide metrics. It's at this point where Control Tower becomes really awesome because we have the concept of an Account Factory. Think of this as a team of robots who are creating, modifying, or deleting AWS accounts as your business needs them. And this can be interacted with both from the Control Tower console or via the Service Catalog.

      Within the custom organizational unit, Account Factory will create AWS accounts in a fully automated way as many of them as you need. The configuration of these accounts is handled by Account Factory. So, from an account and networking perspective, you have baseline or cookie-cutter configurations applied, and this ensures a consistent configuration across all AWS accounts within your Landing Zone. Control Tower utilizes CloudFormation under the covers to implement much of this automation, so expect to see stacks created by the product within your environment. And Control Tower uses both AWS Config and Service Control Policies to implement account guardrails. And these detect drifts away from governance standards, or prevent those drifts from occurring in the first place.

      At a high level, this is how Control Tower looks. Now the product can scale from simple to super complex. This is a product which you need to use in order to really understand. And depending on the course that you're studying, you might have the opportunity to get some hands-on later in the course. If not, don't worry, that means that you only need this high-level understanding for the exam.

      Let's move on and look at the various parts of Control Tower in a little bit more detail. Let's quickly step through the main points at the Landing Zone. It's a feature designed to allow anyone to implement a well-architected, multi-account environment, and it has the concept of a home region, which is the region that you initially deploy the product into, for example, us-east-1. You can explicitly allow or deny the usage of other AWS regions, but the home region, the one that you deploy into, is always available. The Landing Zone is built using AWS Organizations, AWS Config, CloudFormation, and much more. Essentially, Control Tower is a product which brings the features of lots of different AWS products together and orchestrates them.

      I've mentioned that there's a concept of the foundational OU, by default called the Security OU, and within this, Log Archive and Audit AWS accounts. And these are used mainly for security and auditing purposes. You've also got the Sandbox OU which is generally used for testing and less rigid security situations. You can create other organizational units and accounts, and for a real-world deployment of Control Tower, you're generally going to have lots of different organizational units. Potentially, even nested ones to implement a structure which works for your organization.

      Landing Zone utilizes the IAM Identity Center, again, formerly known as AWS SSO, to provide SSO or single sign-on services across multiple AWS accounts within the Landing Zone, and it's also capable of ID Federation. And ID Federation simply means that you can use your existing identity stores to access all of these different AWS accounts. The Landing Zone provides monitoring and notifications using CloudWatch and SNS, and you can also allow end users to provision new AWS accounts within the Landing Zone using Service Catalog.

      This is the Landing Zone at a high level. Let's next talk about guardrails. Guardrails are essentially rules for multi-account governance. Guardrails come in three different types: mandatory, strongly recommended, or elective. Mandatory ones are always applied. Strongly recommended are obviously strongly recommended by AWS. And elective ones can be used to implement fairly niche requirements, and these are completely optional.

      Guardrails themselves function in two different ways. We have preventative, and these stop you doing things within your AWS accounts in your Landing Zone, and these are implemented using Service Control policies, which are part of the AWS Organizations product. These guardrails are either enforced or not enabled, so you can either enforce them or not. And if they're enforced, it simply means that any actions defined by that guardrail are prevented from occurring within any of your AWS accounts. An example of this might be to allow or deny usage of AWS regions, or to disallow bucket policy changes within accounts inside your Landing Zone.

      The second functional type of guardrail is detective, and you can think of this as a compliance check. This uses AWS Config rules and allows you to check that the configuration of a given thing within an AWS account matches what you define as best practice. These type of guardrails are either clear, in violation, or not enabled. And an example of this would be a detective guardrail to check whether CloudTrail is enabled within an AWS account, or whether any EC2 instances have public IPv4 addresses associated with those instances. The important distinction to understand here is that preventative guardrails will stop things occurring, and detective guardrails will only identify those things. So, guardrails are a really important security and governance construct within the Control Tower product.

      Lastly, I want to talk about the Account Factory itself. This is essentially a feature which allows automated account provisioning, and this can be done by either cloud administrators or end users with appropriate permissions. And this automated provisioning includes the application of guardrails, so any guardrails which are defined can be automatically applied to these automatically provisioned AWS accounts.

      Because these accounts can be provisioned by end users, think of these as members of your organization, then either these members of your organization or anyone that you define can be given admin permissions on an AWS account which is automatically provisioned. This allows you to have a truly self-service, automatic process for provisioning AWS accounts so you can allow any member of your organization within tightly controlled parameters to be able to provision accounts for any purpose which you define as okay. And that person will be given admin rights over that AWS account. These can be long-running accounts or short-term accounts. These accounts are also configured with standard account and network configuration. If you have any organizational policies for how networking or any account settings are configured, these automatically provisioned accounts will come with this configuration. And this includes things like the IP addressing used by VPCs within the accounts, which could be automatically configured to avoid things like addressing overlap. And this is really important when you're provisioning accounts at scale.

      The Account Factory allows accounts to be closed or repurposed, and this whole process can be tightly integrated with a business's SDLC or software development life cycle. So, as well as doing this from the console UI, the Control Tower product and Account Factory can be integrated using APIs into any SDLC processes that you have within your organization. If you need accounts to be provisioned as part of a certain stage of application development, or you want accounts to be provisioned as part of maybe client demos or software testing, then you can do this using the Account Factory feature.

      At this point, that is everything I wanted to cover at this high level about Control Tower. If you need practical experience of Control Tower for the course that you are studying, there will be a demo lesson coming up elsewhere in the course, which gives you that practical experience. Don't be concerned if this is the only lesson that there is, or if there's this lesson plus additional deep-dive theory. I'll make sure, for whatever course you're studying, you have enough exposure to Control Tower.

      With that being said, though, that is the end of this high-level video. So go ahead and complete the video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back and welcome to this CloudTrail demo where we're going to set up an organizational trail and configure it to log data for all accounts in our organization to S3 and CloudWatch logs.

      The first step is that you'll need to be logged into the IAM admin user of the management account of the organization. As a reminder, this is the general account. To set up an organizational trail, you always need to be logged into the management account. To set up individual trails, you can do that locally inside each of your accounts, but it's always more efficient to use an organizational trail.

      Now, before we start the demonstration, I want to talk briefly about CloudTrail pricing. I'll make sure this link is in the lesson description, but essentially there is a fairly simple pricing structure to CloudTrail that you need to be aware of.

      The 90-day history that's enabled by default in every AWS account is free. You don't get charged for that; it comes free by default with every AWS account. Next, you have the ability to get one copy of management events free in every region in each AWS account. This means creating one trail that's configured for management events in each region in each AWS account, and that comes for free. If you create any additional trails, so you get any additional copies of management events, they are charged at two dollars per 100,000 events. That won't apply to us in this demonstration, but you need to be aware of that if you're using this in production.

      Logging data events comes at a charge regardless of the number, so we're not going to enable data events for this demo lesson. But if you do enable it, then that comes at a charge of 10 cents per 100,000 events, irrespective of how many trails you have. This charge applies from the first time you're logging any data events.

      What we'll be doing in this demo lesson is setting up an organizational trail which will create a trail in every region in every account inside the organization. But because we get one for free in every region in every account, we won't incur any charges for the CloudTrail side of things. We will be charged for any S3 storage that we use. However, S3 also comes with a free tier allocation for storage, which I don't expect us to breach.

      With that being said, let's get started and implement this solution. To do that, we need to be logged in to the console UI again in the management account of the organization. Then we need to move to the CloudTrail console. If you've been here recently, it will be in the Recently Visited Services. If not, just type CloudTrail in the Find Services box and then open the CloudTrail console.

      Once you're at the console, you might see a screen like this. If you do, then you can just click on the hamburger menu on the left and then go ahead and click on trails. Now, depending on when you're doing this demo, if you see any warnings about a new or old console version, make sure that you select the new version so your console looks like what's on screen now.

      Once you're here, we need to create a trail, so go ahead and click on create trail. To create a trail, you're going to be asked for a few important pieces of information, the first of which is the trail name. For trail name, we're going to use "animals4life.org," so just go ahead and enter that. By default, with this new UI version, when you create a trail, it's going to create it in all AWS regions in your account. If you're logged into the management account of the organization, as we are, you also have the ability to enable it for all regions in all accounts of your organization. We're going to do that because this allows us to have one single logging location for all CloudTrail logs in all regions in all of our accounts, so go ahead and check this box.

      By default, CloudTrail stores all of its logs in an S3 bucket. When you're creating a trail, you have the ability to either create a new S3 bucket to use or you can use an existing bucket. We're going to go ahead and create a brand new bucket for this trail. Bucket names within S3 need to be globally unique, so it needs to be a unique name across all regions and across all AWS accounts. We're going to call this bucket starting with "CloudTrail," then a hyphen, then "animals-for-life," another hyphen, and then you'll need to put a random number. You’ll need to pick something different from me and different from every other student doing this demo. If you get an error about the bucket name being in use, you just need to change this random number.

      You're also able to specify if you want the log files stored in the S3 bucket to be encrypted. This is done using SSE-KMS encryption. This is something that we'll be covering elsewhere in the course, and for production usage, you would definitely want to use it. For this demonstration, to keep things simple, we're not going to encrypt the log files, so go ahead and untick this box.

      Under additional options, you're able to select log file validation, which adds an extra layer of security. This means that if any of the log files are tampered with, you have the ability to determine that. This is a really useful feature if you're performing any account-level audits. In most production situations, I do enable this, but you can also elect to have an SNS notification delivery. So, every time log files are delivered into this S3 bucket, you can have a notification. This is useful for production usage or if you need to integrate this with any non-AWS systems, but for this demonstration, we'll leave this one unchecked.

      You also have the ability, as well as storing these log files into S3, to store them in CloudWatch logs. This gives you extra functionality because it allows you to perform searches, look at the logs from a historical context inside the CloudWatch logs user interface, as well as define event-driven processes. You can configure CloudWatch logs to scan these CloudTrail logs and, in the event that any particular piece of text occurs in the logs (e.g., any API call, any actions by a user), you can generate an event that can invoke, for example, a Lambda function or spawn some other event-driven processing. Don't worry if you don't understand exactly what this means at this point; I'll be talking about all of this functionality in detail elsewhere in the course. For this demonstration, we are going to enable CloudTrail to put these logs into CloudWatch logs as well, so check this box. You can choose a log group name within CloudWatch logs for these CloudTrail logs. If you want to customize this, you can, but we're going to leave it as the default.

      As with everything inside AWS, if a service is acting on our behalf, we need to give it the permissions to interact with other AWS services, and CloudTrail is no exception. We need to give CloudTrail the ability to interact with CloudWatch logs, and we do that using an IAM role. Don’t worry, we’ll be talking about IAM roles in detail elsewhere in the course. For this demonstration, just go ahead and select "new" because we're going to create a new IAM role that will give CloudTrail the ability to enter data into CloudWatch logs.

      Now we need to provide a role name, so go ahead and enter "CloudTrail_role_for_CloudWatch_logs" and then an underscore and then "animals_for_life." The name doesn’t really matter, but in production settings, you'll want to make sure that you're able to determine what these roles are for, so we’ll use a standard naming format. If you expand the policy document, you'll be able to see the exact policy document or IAM policy document that will be used to give this role the permissions to interact with CloudWatch logs. Don’t worry if you don’t fully understand policy documents at this point; we’ll be using them throughout the course, and over time you'll become much more comfortable with exactly how they're used. At a high level, this policy document will be attached to this role, and this is what will give CloudTrail the ability to interact with CloudWatch logs.

      At this point, just scroll down; that's everything that we need to do, so go ahead and click on "next." Now, you'll need to select what type of events you want this trail to log. You’ve got three different choices. The default is to log only management events, so this logs any events against the account or AWS resources (e.g., starting or stopping an EC2 instance, creating or deleting an EBS volume). You've also got data events, which give you the ability to log any actions against things inside resources. Currently, CloudTrail supports a wide range of services for data event logging. For this demonstration, we won't be setting this up with data events initially because I’ll be covering this elsewhere in the course. So, go back to the top and uncheck data events.

      You also have the ability to log insight events, which can identify any unusual activity, errors, or user behavior on your account. This is especially useful from a security perspective. For this demonstration, we won’t be logging any insight events; we’re just going to log management events. For management events, you can further filter down to read or write or both and optionally exclude KMS or RDS data API events. For this demo lesson, we’re just going to leave it as default, so make sure that read and write are checked. Once you've done that, go ahead and click on "next." On this screen, just review everything. If it all looks good, click on "create trail."

      Now, if you get an error saying the S3 bucket already exists, you'll just need to choose a new bucket name. Click on "edit" at the top, change the bucket name to something that's globally unique, and then follow that process through again and create the trail.

      Certainly! Here is the continuation and completion of the transcript:


      After a few moments, the trail will be created. It should say "US East Northern Virginia" as the home region. Even though you didn't get the option to select it because it's selected by default, it is a multi-region trail. Finally, it is an organizational trail, which means that this trail is now logging any CloudTrail events from all regions in all accounts in this AWS organization.

      Now, this isn't real-time, and when you first enable it, it can take some time for anything to start to appear in either S3 or CloudWatch logs. At this stage, I recommend that you pause the video and wait for 10 to 15 minutes before continuing, because the initial delivery of that first set of log files through to S3 can take some time. So pause the video, wait 10 to 15 minutes, and then you can resume.

      Next, right-click the link under the S3 bucket and open that in a new tab. Go to that tab, and you should start to see a folder structure being created inside the S3 bucket. Let's move down through this folder structure, starting with CloudTrail. Go to US East 1 and continue down through this folder structure.

      In my case, I have quite a few of these log files that have been delivered already. I'm going to pick one of them, the most recent, and just click on Open. Depending on the browser that you're using, you might have to download and then uncompress this file. Because I'm using Firefox, it can natively open the GZ compressed file and then automatically open the JSON log file inside it.

      So this is an example of a CloudTrail event. We're able to see the user identity that actually generates this event. In this case, it's me, I am admin. We can see the account ID that this event is for. We can see the event source, the event name, the region, the source IP address, the user agent (in this case, the console), and all of the relevant information for this particular interaction with the AWS APIs are logged inside this CloudTrail event.

      Don’t worry if this doesn’t make a lot of sense at this point. You’ll get plenty of opportunities to interact with this type of logging event as you go through the various theory and practical lessons within the course. For now, I just want to highlight exactly what to expect with CloudTrail logs.

      Since we’ve enabled all of this logging information to also go into CloudWatch logs, we can take a look at that as well. So back at the CloudTrail console, if we click on Services and then type CloudWatch, wait for it to pop up, locate Logs underneath CloudWatch, and then open that in a new tab.

      Inside CloudWatch, on the left-hand menu, look for Logs, and then Log Groups, and open that. You might need to give this a short while to populate, but once it does, you should see a log group for the CloudTrail that you’ve just created. Go ahead and open that log group.

      Inside it, you’ll see a number of log streams. These log streams will start with your unique organizational code, which will be different for you. Then there will be the account number of the account that it represents. Again, these will be different for you. And then there’ll be the region name. Because I’m only interacting with the Northern Virginia region, currently, the only ones that I see are for US East 1.

      In this particular account that I’m in, the general account of the organization, if I look at the ARN (Amazon Resource Name) at the top or after US East 1 here, this number is my account number. This is the account number of my general account. So if I look at the log streams, you’ll be able to see that this account (the general account) matches this particular log stream. You’ll be able to do the same thing in your account. If you look for this account ID and then match it with one of the log streams, you'll be able to pull the logs for the general AWS account.

      If I go inside this particular log stream, as CloudTrail logs any activity in this account, all of that information will be populated into CloudWatch logs. And that’s what I can see here. If I expand one of these log entries, we’ll see the same formatted CloudTrail event that I just showed you in my text editor. So the only difference when using CloudWatch logs is that the CloudTrail events also get entered into a log stream in a log group within CloudWatch logs. The format looks very similar.

      Returning to the CloudTrail console, one last thing I want to highlight: if you expand the menu on the left, whether you enable a particular trail or not, you’ve always got access to the event history. The event history stores a log of all CloudTrail events for the last 90 days for this particular account, even if you don’t have a specific trail enabled. This is standard functionality. What a trail allows you to do is customize exactly what happens to that data. This area of the console, the event history, is always useful if you want to search for a particular event, maybe check who’s logged onto the account recently, or look at exactly what the IAM admin user has been doing within this particular AWS account.

      The reason why we created a trail is to persistently store that data in S3 as well as put it into CloudWatch logs, which gives us that extra functionality. With that being said, that’s everything I wanted to cover in this demo lesson.

      One thing you need to be aware of is that S3, as a service, provides a certain amount of resource under the free tier available in every new AWS account, so you can store a certain amount of data in S3 free of charge. The problem with CloudTrail, and especially organizational trails, is that they generate quite a large number of requests. There is also, in addition to space, a number of requests per month that are part of the free tier.

      If you leave this CloudTrail enabled for the duration of your studies, for the entire month, it is possible that this will go slightly over the free tier allocation for requests within the S3 service. You might see warnings that you’re approaching a billable threshold, and you might even get a couple of cents of bill per month if you leave this enabled all the time. To avoid that, if you just go to Trails, open up the trail that you’ve created, and then click on Stop Logging. You’ll need to confirm that by clicking on Stop Logging, and at that point, no logging will occur into the S3 bucket or into CloudWatch logs, and you won’t experience those charges.

      For any production usage, the low cost of this service means that you would normally leave it enabled in all situations. But to keep costs within the free tier for this course, you can, if required, just go ahead and stop the logging. If you don’t mind a few cents per month of S3 charges for CloudTrail, then by all means, go ahead and leave it enabled.

      With that being said, that’s everything I wanted to cover in this demo lesson. So go ahead, complete the lesson, and when you're ready, I look forward to you joining me in the next.

    1. Welcome to this lesson, where I'm going to be introducing CloudTrail.

      CloudTrail is a product that logs API actions which affect AWS accounts. If you stop an instance, that's logged. If you change a security group, that's logged too. If you create or delete an S3 bucket, that's logged by CloudTrail. Almost everything that can be done to an AWS account is logged by this product.

      Now, I want to quickly start with the CloudTrail basics. The product logs API calls or account activities, and every one of those logged activities is called a CloudTrail event. A CloudTrail event is a record of an activity in an AWS account. This activity can be an action taken by a user, a role, or a service.

      CloudTrail by default stores the last 90 days of CloudTrail events in the CloudTrail event history. This is an area of CloudTrail which is enabled by default in AWS accounts. It's available at no cost and provides 90 days of history on an AWS account.

      If you want to customize CloudTrail in any way beyond this 90-day event history, you need to create a trail. We'll be looking at the architecture of a trail in a few moments' time.

      CloudTrail events can be one of three different types: management events, data events, and insight events. If applicable to the course you are studying, I'll be talking about insight events in a separate video. For now, we're going to focus on management events and data events.

      Management events provide information about management operations performed on resources in your AWS account. These are also known as control plane operations. Think of things like creating an EC2 instance, terminating an EC2 instance, creating a VPC. These are all control plane operations.

      Data events contain information about resource operations performed on or in a resource. Examples of this might be objects being uploaded to S3 or objects being accessed from S3, or when a Lambda function is invoked. By default, CloudTrail only logs management events because data events are often much higher volume. Imagine if every access to an S3 object was logged; it could add up pretty quickly.

      A CloudTrail trail is the unit of configuration within the CloudTrail product. It's a way you provide configuration to CloudTrail on how to operate. A trail logs events for the AWS region that it's created in. That's critical to understand. CloudTrail is a regional service.

      When you create a trail, it can be configured to operate in one of two ways: as a one-region trail or as an all-regions trail. A single-region trail is only ever in the region that it's created in, and it only logs events for that region. An all-regions trail, on the other hand, can be thought of as a collection of trails in every AWS region, but it's managed as one logical trail. It also has the additional benefit that if AWS adds any new regions, the all-regions trail is automatically updated.

      This is a specific configuration item on a trail which determines if it only logs events for the region that it's in or if it also logs global services events. Most services log events in the region where the event occurred. For example, if you create an EC2 instance in AP Southeast 2, it’s logged to that region. A trail would need to be either a one-region trail in that region or an all-regions trail to capture that event.

      A very small number of services log events globally to one region. For example, global services such as IAM, STS, or CloudFront are very globally-focused services and always log their events to US East 1, which is Northern Virginia. These types of events are called global service events, and a trail needs to have this enabled in order to log these events. This feature is normally enabled by default if you create a trail inside the user interface.

      AWS services are largely split up into regional services and global services. When these different types of services log to CloudTrail, they either log in the region that the event is generated in or they log to US East 1 if they are global services. So, when you're diagnosing problems or architecting solutions, if the logs you are trying to reach are generated by global services like IAM, STS, or CloudFront, these will be classified as global service events and that will need to be enabled on a trail.

      Otherwise, a trail will only log events for the isolated region that it’s created in. When you create a trail, it is one of two types: one-region or all-regions. A one-region trail is always isolated to that one region, and you would need to create one-region trails in every region if you wanted to do it manually. Alternatively, you could create an all-regions trail, which encompasses all of the regions in AWS and is automatically updated as AWS adds new regions.

      Once you’ve created a trail, management events and data events are all captured by the trail based on whether it's isolated to a region or set to all regions. For an all-region trail, it captures management events and, if enabled, data events. Data events are not generally enabled by default and must be explicitly set when creating a trail. This trail will then listen to everything that's occurring in the account.

      Remember that the CloudTrail event history is limited to 90 days. However, when you create a trail, you can be much more flexible. A trail by default can store the events in a definable S3 bucket, and the logs generated and stored in an S3 bucket can be stored there indefinitely. You are only charged for the storage used in S3. These logs are stored as a set of compressed JSON log files, which consume minimal space. Being JSON formatted, they can be read by any tooling capable of reading standard format files, which is a great feature of CloudTrail.

      Another option is that CloudTrail can be integrated with CloudWatch Logs, allowing data to be stored in that product. CloudTrail can take all the logging data it generates and, in addition to putting it into S3, it can also put it into CloudWatch Logs. Once it's in CloudWatch Logs, you can use that product to search through it or use a metric filter to take advantage of the data stored there. This makes it much more powerful and gives you access to many more features if you use CloudWatch Logs versus S3.

      One of the more recent additions to the CloudTrail product is the ability to create an organizational trail. If you create this trail from the management account of an organization, it can store all the information for all the accounts inside that organization. This provides a single management point for all API and account events across every account in the organization, which is super powerful and makes managing multi-account environments much easier.

      So, we need to talk through some important elements of CloudTrail point by point. CloudTrail is enabled by default on AWS accounts, but it’s only the 90-day event history that’s enabled by default. You don’t get any storage in S3 unless you configure a trail. Trails are how you can take the data that CloudTrail’s got access to and store it in better places, such as S3 and CloudWatch Logs.

      The default for trails is to store management events only, which includes management plane events like creating an instance, stopping an instance, terminating an instance, creating or deleting S3 buckets, and logins to the console. Anything interacting with AWS products and services from a management perspective is logged by default in CloudTrail. Data events need to be specifically enabled and come at an extra cost. I’ll discuss this in more detail in the demo lesson, as you need to be aware of the pricing of CloudTrail. Much of the service is free, but there are certain elements that do carry a cost, especially if you use it in production.

      Most AWS services log data to the same region that the service is in. There are a few specific services, such as IAM, STS, and CloudFront, which are classified as true global services and log their data as global service events to US East 1. A trail needs to be enabled to capture that data.

      That’s critical and might come up as an exam question. What you will also definitely find coming up as an exam-style question is where to use CloudTrail for real-time logging. This is one of the limitations of the product—it is not real-time. CloudTrail typically delivers log files within 15 minutes of the account activity occurring and generally publishes log files multiple times per hour. This means it's not real-time; you can't rely on CloudTrail to provide a complete and exhaustive list of events up to the very point you're looking. Sometimes, it takes a few minutes for the data to arrive in S3 or CloudWatch Logs. Keep this in mind if you face any exam questions about real-time logging—CloudTrail is not the product.

      Okay, so that's the end of the theory in this lesson. It's time for a demo. In the next lesson, we’ll be setting up an organizational trail within our AWS account structure. We’ll configure it to capture all the data for all our member accounts and our management account, storing this data in an S3 bucket and CloudWatch Logs within the management account. I can’t wait to get started. It’s a fun one and will prove very useful for both the exam and real-world usage.

      So go ahead, complete this video, and when you're ready, you can join me in the demo lesson.

    1. Welcome to this lesson, where I'm going to introduce the theory and architecture of CloudWatch Logs.

      I've already covered the metrics side of CloudWatch earlier in the course, and I'm covering the logs part now because you'll be using it when we cover CloudTrail. In the CloudTrail demo, we'll be setting up CloudTrail and using CloudWatch Logs as a destination for those logs. So, you'll need to understand it, and we'll be covering the architecture in this lesson. Let's jump in and get started.

      CloudWatch Logs is a public service. The endpoint to which applications connect is hosted in the AWS public zone. This means you can use the product within AWS VPCs, from on-premises environments, and even other cloud platforms, assuming that you have network connectivity as well as AWS permissions.

      The CloudWatch Logs product allows you to store, monitor, and access logging data. Logging data, at a very basic level, consists of a piece of information, data, and a timestamp. The timestamp generally includes the year, month, day, hour, minute, second, and timezone. There can be more fields, but at a minimum, it's generally a timestamp and some data.

      CloudWatch Logs has built-in integrations with many AWS services, including EC2, VPC Flow Logs, Lambda, CloudTrail, Route 53, and many more. Any services that integrate with CloudWatch Logs can store data directly inside the product. Security for this is generally provided by using IAM roles or service roles.

      For anything outside AWS, such as logging custom application or OS logs on EC2, you can use the unified CloudWatch agent. I’ve mentioned this before and will be demoing it later in the EC2 section of the course. This is how anything outside of AWS products and services can log data into CloudWatch Logs. So, it’s either AWS service integrations or the unified CloudWatch agent. There is a third way, using development kits for AWS to implement logging into CloudWatch Logs directly into your application, but that tends to be covered in developer and DevOps AWS courses. For now, just remember either AWS service integrations or the unified CloudWatch agent.

      CloudWatch Logs are also capable of taking logging data and generating a metric from it, known as a metric filter. Imagine a situation where you have a Linux instance, and one of the operating system log files logs any failed connection attempts via SSH. If this logging information was injected into CloudWatch Logs, a metric filter can scan those logs constantly. Anytime it sees a mention of the failed SSH connection, it can increment a metric within CloudWatch. You can then have alarms based on that metric, and I’ll be demoing that very thing later in the course.

      Let’s look at the architecture visually because I'll be showing you how this works in practice in the CloudTrail demo, which will be coming up later in the section. Architecturally, CloudWatch Logs looks like this: It’s a regional service. So, for this example, let’s assume we’re talking about us-east-1.

      The starting point is our logging sources, which can include AWS products and services, mobile or server-based applications, external compute services (virtual or physical servers), databases, or even external APIs. These sources inject data into CloudWatch Logs as log events.

      Log events consist of a timestamp and a message block. CloudWatch Logs treats this message as a raw block of data. It can be anything you want, but there are ways the data can be interpreted, with fields and columns defined. Log events are stored inside log streams, which are essentially a sequence of log events from the same source.

      For example, if you had a log file stored on multiple EC2 instances that you wanted to inject into CloudWatch Logs, each log stream would represent the log file for one instance. So, you’d have one log stream for instance one and one log stream for instance two. Each log stream is an ordered set of log events for a specific source.

      We also have log groups, which are containers for multiple log streams of the same type of logging. Continuing the example, we would have one log group containing everything for that log file. Inside this log group would be different log streams, each representing one source. Each log stream is a collection of log events. Every time an item was added to the log file on a single EC2 instance, there would be one log event inside one log stream for that instance.

      A log group also stores configuration settings, such as retention settings and permissions. When we define these settings on a log group, they apply to all log streams within that log group. It’s also where metric filters are defined. These filters constantly review any log events for any log streams in that log group, looking for certain patterns, such as an application error code or a failed SSH login. When detected, these metric filters increment a metric, and metrics can have associated alarms. These alarms can notify administrators or integrate with AWS or external systems to take action.

      CloudWatch Logs is a powerful product. This is the high-level architecture, but don’t worry—you’ll get plenty of exposure to it throughout the course because many AWS products integrate with CloudWatch Logs and use it to store their logging data. We’ll be coming back to this product time and again as we progress through the course. CloudTrail uses CloudWatch Logs, Lambda uses CloudWatch Logs, and VPC Flow Logs use CloudWatch Logs. There are many examples of AWS products where we’ll be integrating them with CloudWatch Logs.

      I just wanted to introduce it at this early stage of the course. That’s everything I wanted to cover in this theory lesson. Thanks for watching. Go ahead, complete this video, and when you’re ready, join me in the next.

    1. Welcome back, and in this demo lesson, I want to give you some experience working with Service Control Policies (SCPs).

      At this point, you've created the AWS account structure which you'll be using for the remainder of the course. You've set up an AWS organization, with the general account that created it becoming the management account. Additionally, you've invited the production AWS account into the organization and created the development account within it.

      In this demo lesson, I want to show you how you can use SCPs to restrict what identities within an AWS account can do. This is a feature of AWS Organizations.

      Before we dive in, let's tidy up the AWS organization. Make sure you're logged into the general account, the management account of the organization, and then navigate to the organization's console. You can either type that into the 'Find Services' box or select it from 'Recently Used Services.'

      As discussed in previous lessons, AWS Organizations allows you to organize accounts with a hierarchical structure. Currently, there's only the root container of the organization. To create a hierarchical structure, we need to add some organizational units. We will create a development organizational unit and a production organizational unit.

      Select the root container at the top of the organizational structure. Click on "Actions" and then "Create New." For the production organizational unit, name it 'prod.' Scroll down and click on "Create Organizational Unit." Next, do the same for the development unit: select 'Route,' click on "Actions," and then "Create New." Under 'Name,' type 'dev,' scroll down, and click on "Create Organizational Unit."

      Now, we need to move our AWS accounts into these relevant organizational units. Currently, the Development, Production, and General accounts are all contained in the root container, which is the topmost point of our hierarchical structure.

      To move the accounts, select the Production AWS account, click on "Actions," and then "Move." In the dialogue that appears, select the Production Organizational Unit and click "Move." Repeat this process for the Development AWS account: select the Development AWS account, click "Actions," then "Move," and select the 'dev' OU before clicking "Move."

      Now, we've successfully moved the two AWS accounts into their respective organizational units. If you select each organizational unit in turn, you can see that 'prod' contains the production AWS account, and 'dev' contains the development AWS account. This simple hierarchical structure is now in place.

      To prepare for the demo part of this lesson where we look at SCPs, move back to the AWS console. Click on AWS, then the account dropdown, and switch roles into the production AWS account by selecting 'Prod' from 'Role History.'

      Once you're in the production account, create an S3 bucket. Type S3 into the 'Find Services' box or find it in 'Recently Used Services' and navigate to the S3 console. Click on "Create Bucket." For the bucket name, call it 'CatPics' followed by a random number—S3 bucket names must be globally unique. I’ll use 1, lots of 3s, and then 7. Ensure you select the US East 1 region for the bucket. Scroll down and click "Create Bucket."

      After creating the bucket, go inside it and upload some files. Click on "Add Files," then download the cat picture linked to this lesson to your local machine. Upload this cat picture to the S3 bucket by selecting it and clicking "Open," then "Upload" to complete the process.

      Once the upload finishes, you can view the picture of Samson. Click on it to see Samson looking pretty sleepy. This demonstrates that you can currently access the Samson.jpg object while operating within the production AWS account.

      The key point here is that you’ve assumed an IAM role. By switching roles into the production account, you’ve assumed the role called "organization account access role," which has the administrator access managed policy attached.

      Now, we’ll demonstrate how this can be restricted using SCPs. Move back to the main AWS console. Click on the account dropdown and switch back to the general AWS account. Navigate to AWS Organizations, then Policies. Currently, most options are disabled, including Service Control Policies, Tag Policies, AI Services, Opt-out Policies, and Backup Policies.

      Click on Service Control Policies and then "Enable" to activate this functionality. This action adds the "Full AWS Access" policy to the entire organization, which imposes no restrictions, so all AWS accounts maintain full access to all AWS services.

      To create our own service control policy, download the file named DenyS3.json linked to this lesson and open it in a code editor. This SCP contains two statements. The first statement is an allow statement with an effect of allow, action as star (wildcard), and resource as star (wildcard). This replicates the full AWS access SCP applied by default. The second statement is a deny statement that denies any S3 actions on any AWS resource. This explicit deny overrides the explicit allow for S3 actions, resulting in access to all AWS services except S3.

      Copy the content of the DenyS3.json file into your clipboard. Move back to the AWS console, go to the policy section, and select Service Control Policies. Click "Create Policy," delete the existing JSON in the policy box, and paste the copied content. Name this policy "Allow all except S3" and create it.

      Now, go to AWS Accounts on the left menu, select the prod OU, and click on the Policies tab. Attach the new policy "Allow all except S3" by clicking "Attach" in the applied policies box. We will also detach the full AWS access policy directly attached. Check the box next to full AWS access, click "Detach," and confirm by clicking "Detach Policy."

      Now, the only service control policy directly attached to production is "Allow all except S3," which allows access to all AWS products and services except S3.

      To verify, go back to the main AWS console and switch roles into the production AWS account. Go to the S3 console and you should receive a permissions error, indicating that you don't have access to list buckets. This is because the SCP attached to the production account explicitly denies S3 access. Access to other services remains unaffected, so you can still interact with EC2.

      If we switch back to the general account, reattach the full AWS access policy, and detach "Allow all except S3," the production account will regain access to S3. By following the same process, you’ll be able to access the S3 bucket and view the object once again.

      This illustrates how SCPs can be used to restrict access for identities within an AWS account, in this case, the production AWS account.

      To clean up, delete the bucket. Select the catpics bucket, click "Empty," type "permanently delete," and select "Empty." Once that's done, you can delete the bucket by selecting it, clicking "Delete," confirming the bucket name, and then clicking "Delete Bucket."

      You’ve now demonstrated full control over S3, evidenced by successfully deleting the bucket. This concludes the demo lesson. You’ve created and applied an SCP that restricts S3 access, observed its effects, and cleaned up. We’ll discuss more about boundaries and restrictions in future lessons. For now, complete this video, and I'll look forward to seeing you in the next lesson.

    1. Welcome back, and in this lesson, I'll be talking about service control policies, or SCPs. SCPs are a feature of AWS Organizations which can be used to restrict AWS accounts. They're an essential feature to understand if you are involved in the design and implementation of larger AWS platforms. We've got a lot to cover, so let's jump in and get started.

      At this point, this is what our AWS account setup looks like. We've created an organization for Animals4life, and inside it, we have the general account, which from now on I'll be referring to as the management account, and then two member accounts, so production, which we'll call prod, and development, which we'll be calling dev. All of these AWS accounts are within the root container of the organization. That's to say they aren't inside any organizational units. In the next demo lesson, we're going to be adding organizational units, one for production and one for development, and we'll be putting the member accounts inside their respective organizational units.

      Now, let's talk about service control policies. The concept of a service control policy is simple enough. It's a policy document, a JSON document, and these service control policies can be attached to the organization as a whole by attaching them to the root container, or they can be attached to one or more organizational units. Lastly, they can even be attached to individual AWS accounts. Service control policies inherit down the organization tree. This means if they're attached to the organization as a whole, so the root container of the organization, then they affect all of the accounts inside the organization. If they're attached to an organizational unit, then they impact all accounts directly inside that organizational unit, as well as all accounts within OUs inside that organizational unit. If you have nested organizational units, then by attaching them to one OU, they affect that OU and everything below it. If you attach service control policies to one or more accounts, then they just directly affect those accounts that they're attached to.

      Now, I mentioned in an earlier lesson that the management account of an organization is special. One of the reasons it's special is that even if the management account has service control policies attached, either directly via an organizational unit, or on the root container of the organization itself, the management account is never affected by service control policies. This can be both beneficial and it can be a limitation, but as a minimum, you need to be aware of it as a security practice. Because the management account can't be restricted using service control policies, I generally avoid using the management account for any AWS resources. It's the only AWS account within AWS Organizations which can't be restricted using service control policies. As a takeaway, just remember that the management account is special and it's unaffected by any service control policies, which are attached to that account either directly or indirectly.

      Now, service control policies are account permissions boundaries. What I mean by that is they limit what the AWS account can do, including the Account Root User within that account. I talked earlier in the course about how you can't restrict an Account Root User. And that is true. You can't directly restrict what the Account Root User of an AWS account can do. The Account Root User always has full permissions over that entire AWS account, but with a service control policy, you're actually restricting what the account itself can do, specifically any identities within that account. So you're indirectly restricting the Account Root User because you're reducing the allowed permissions on the account; you're also reducing what the effective permissions on the Account Root User are. This is a really fine detail to understand. You can never restrict the Account Root User. It will always have 100% access to the account, but if you restrict the account, then in effect, you're also restricting the Account Root User.

      Now, you might apply a service control policy to prevent any usage of that account outside a known region, for example, us-east-1. You might also apply a service control policy which only allows a certain size of EC2 instance to be used within the account. Service control policies are a really powerful feature for any larger, more complex AWS deployments. The critical thing to understand about service control policies is they don't grant any permissions. Service control policies are just a boundary. They define the limit of what is and isn't allowed within the account, but they don't grant permissions. You still need to give identities within that AWS account permissions to AWS resources, but any SCPs will limit the permissions that can be assigned to individual identities.

      You can use service control policies in two ways. You can block by default and allow certain services, which is an allow list. Or you can allow by default and block access to certain services, which is a deny list. The default is a deny list. When you enable SCPs on your organization, AWS applies a default policy called full AWS access. This is applied to the organization and all OUs within that organization. This policy means that in the default implementation, service control policies have no effect since nothing is restricted. As a reminder, service control policies don't grant permissions, but when SCPs are enabled, there is an implicit default deny, just like IAM policies. If you had no initial allow, then everything would be denied. So the default is this full access policy, which essentially means no restrictions. It has the effect of making SCPs a deny list architecture, so you need to add any restrictions that you want to any AWS accounts within the organization. An example is that you could add another policy, such as this one, called DenyS3. This adds a deny policy for the entire S3 set of API operations, effectively denying S3. You need to remember that SCPs don't actually grant any access rights, but they establish which permissions can be granted in an account. The same priority rules apply: deny, allow, deny. Anything explicitly allowed in an SCP is a service which can have access granted to identities within that account, unless there's an explicit deny within an SCP, then a service cannot be granted. Explicit deny always wins. And in the absence of either, if we didn't have this full AWS access policy in place, then there would be an implicit deny, which blocks access to everything.

      The benefit of using deny lists is that because your foundation is to allow wildcard access, so all actions on all resources, as AWS extends the amounts of products and services which are available inside the platform, this allow list constantly expands to cover those services, so it's fairly low admin overhead. You simply need to add any services which you want to deny access to via an explicit deny. In certain situations, you might need to be more conscious about usage in your accounts, and that's where you'd use allow lists. To implement allow lists, it's a two-part architecture. One part of it is to remove the AWS full access policy. This means that only the implicit default deny is in place and active, and then you would need to add any services which you want to allow into a new policy. In this case, S3 and EC2. So in this architecture, we wouldn't have this full AWS access. We would be explicitly allowing S3 and EC2 access. So no matter what identity permissions identities in this account are provided with, they would only ever be allowed to access S3 and EC2. This is more secure because you have to explicitly say which services can be allowed access for users in those accounts, but it's much easier to make a mistake and block access to services which you didn't intend to. It's also much more admin overhead because you have to add services as your business requirements dictate. You can't simply have access to everything and deny services you don't want access to. With this type of architecture, you have to explicitly add each and every service which you want identities within the account to be able to access. Generally, I would normally suggest using a deny list architecture because, simply put, it's much lower admin overhead.

      Before we go into a demo, I want to visually show you how SCPs affect permissions. This is visually how SCPs impact permissions within an AWS account. In the left orange circle, this represents the different services that have been granted access to identities in an account using identity policies. On the right in red, this represents which services an SCP allows access to. So the SCP states that the three services in the middle and the service on the right are allowed access as far as the SCP is concerned, and the identity policies which were applied to identities within the account, so the orange circle on the left, grant access to four different services: the three in the middle and the one on the left.

      Only permissions which are allowed within identity policies in the account and are allowed by a service control policy are actually active. On the right, this access permission has no effect because while it's allowed within an SCP, an SCP doesn't grant access to anything; it just controls what can and can't be allowed by identity policies within that account. Because no identity policy allows access to this resource, then it has no effect. On the left, this particular access permission is allowed within an identity policy, but it's not effectively allowed because it's not allowed within an SCP. So only things which are involved, the identity policy and an SCP, are actually allowed. In this case, this particular access permission on the left has no effect because it's not within a service control policy, so it's denied.

      At an associate level, this is what you need to know for the exam. It's just simply understanding that your effective permissions for identities within an account are the overlap between any identity policies and any applicable SCPs. This is going to make more sense if you experience it with a demo, so this is what we're going to do next. Now that you've set up the AWS organization for the Animals4life business, it's time to put some of this into action. So I'm going to finish this lesson here and then in the next lesson, which is a demo, we're going to continue with the practical part of implementing SCPs. So go ahead and complete this video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back! In this demo lesson, you're going to create the AWS account structure which you'll use for the remainder of the course. At this point, you need to log in to the general AWS account. I’m currently logged in as the IAM admin user of my general AWS account, with the Northern Virginia region selected.

      You’ll need either two different web browsers or a single web browser like Firefox that supports different sessions because we’ll be logged into multiple AWS accounts at once. The first task is to create the AWS organization. Since I'm logged in to a standard AWS account that isn’t part of an AWS organization, it’s neither a management account nor a member account. We need to move to the AWS Organizations part of the console and create the organization.

      To start, go to "Find Services," type "Organizations," and click to move to the AWS Organizations console. Once there, click "Create Organization." This will begin the process of creating the AWS organization and convert the standard account into the management account of the organization. Click on "Create Organization" to complete the process. Now, the general account is the management account of the AWS organization.

      You might see a message indicating that a verification email has been sent to the email address associated with the general AWS account. Click the link in that email to verify the address and continue using AWS Organizations. If you see this notification, verify the email before proceeding. If not, you can continue.

      Now, open a new web browser or a browser session like Firefox and log in to the production AWS account. Ensure this is a separate session; if unsure, use a different browser to maintain logins to both the management and production accounts. I’ll log in to the IAM admin user of the production AWS account.

      With the production AWS account logged in via a separate browser session, copy the account ID for the production AWS account from the account dropdown. Then, return to the browser session with the general account, which is now the management account of the organization. We’ll invite the production AWS account into this organization.

      Click on "Add Account," then "Invite Account." Enter either the email address used while signing up or the account ID of the production account. I’ll enter the account ID. If you’re inviting an account you administer, no notes are needed. However, if the account is administered by someone else, you may include a message. After entering the email or account ID, scroll down and click "Send Invitation."

      Depending on your AWS account, you might receive an error message about too many accounts within the organization. If so, log a support request to increase the number of allowed accounts. If no error message appears, the invite process has begun.

      Next, accept the invite from the production AWS account. Go back to the tab with the general AWS account, move to the Organizations console, and click "Invitations" on the middle left. You should see an overview of all invitations related to the production AWS account. Click "Accept" to complete the process of joining the organization. Now, the production account is a member of the AWS organization.

      To verify, return to the general account tab and refresh. You should now see two AWS accounts: the general and the production accounts. Next, I’ll demonstrate how to role switch into the production AWS account, now a member of the organization.

      When adding an account to an organization, you can either invite an existing account or create a new one within the organization. If creating a new account, a role is automatically created for role switching. If inviting an existing account, you need to manually add this role.

      To do this, switch to the browser or session where you're logged into the production AWS account. Click on the services search box, type IAM, and move to the IAM console to create IAM roles. Click on "Create Role," select "Another AWS Account," and enter the account ID of the general AWS account, which is now the management account.

      Copy the account ID of the general AWS account into the account ID box, then click "Next." Attach the "AdministratorAccess" policy to this role. On the next screen, name the role "OrganizationAccountAccessRole" with uppercase O, A, A, and R, and note that "Organization" uses the U.S. spelling with a Z. Click "Create Role."

      In the role details, select "Trust Relationships" to verify that the role trusts the account ID of your general AWS account, which allows identities within the general account to assume this role.

      Next, switch back to the general AWS account. Copy the account ID for the production AWS account because we will switch into it using role switch. In the AWS console, click on the account dropdown and select "Switch Roles." Paste the production account ID into the account ID box, and enter the role name "OrganizationAccountAccessRole" with uppercase O, A, A, and R.

      For the display name, use "Prod" for production, and pick red as the color for easy identification. Click "Switch Role" to switch into the production AWS account. You’ll see the red color and "Prod" display name indicating a successful switch.

      To switch back to the general account, click on "Switch Back." In the role history section, you can see shortcuts for switching roles. Click "Prod" to switch back to the production AWS account using temporary credentials granted by the assumed role.

      Now, let’s create the development AWS account within our organization. Close the browser window or tab with the production AWS account as it’s no longer needed. Return to the AWS Organizations console, click "Add Account," and then "Create Account." Name the account "Development," following the same naming structure used for general and production accounts.

      Provide a unique email address for the development AWS account. Use the same email structure you’ve used for previous accounts, such as "Adrian+TrainingAWSDevelopment" for consistency.

      In the box for the role name, use "OrganizationAccountAccessRole" with uppercase O, A, A, and R, and the U.S. spelling. Click "Create" to create the development account. If you encounter an error about too many accounts, you might need to request an increase in the account limit.

      The development account will be created within the organization, and this may take a few minutes. Refresh to see the new development account with its own account ID. Copy this account ID for the switch role dialogue.

      Click on the account dropdown, select "Switch Roles," and enter the new development account ID. For the role name, use "OrganizationAccountAccessRole" and for the display name, use "Dev" for development with yellow as the color for distinction. Click "Switch Role" to switch into the development AWS account.

      In the AWS console, you’ll see the new development account. You can switch directly between the general, production, and development accounts using role switch shortcuts. AWS automatically created the "OrganizationAccountAccessRole" in the development account.

      In summary, you now have three AWS accounts: the general AWS account (management account), the production AWS account, and the development AWS account. This completes the account structure for the course. Complete this video, and I'll look forward to seeing you in the next lesson.

    1. Welcome to this lesson, where I'll be introducing AWS Organizations. AWS Organizations is a product that allows larger businesses to manage multiple AWS accounts in a cost-effective way with little to no management overhead.

      Organizations is a product that has evolved significantly over the past few years, and it's worthwhile to step through that evolution to understand all of its different features. We’ve got a lot to cover, so let's jump in and get started.

      Without AWS Organizations, many large businesses would face the challenge of managing numerous AWS accounts. In the example onscreen, there are four accounts, but I've worked with some larger enterprises with hundreds of accounts and have heard of even more. Without AWS Organizations, each of these accounts would have its own pool of IAM users as well as separate payment methods. Beyond 5 to 10 accounts, this setup becomes unwieldy very quickly.

      AWS Organizations is a simple product to understand. You start with a single AWS account, which I'll refer to as a standard AWS account from now on. A standard AWS account is an AWS account that is not part of an organization. Using this standard AWS account, you create an AWS Organization.

      It’s important to understand that the organization isn't created within this account; you're simply using the account to create the organization. This standard AWS account that you use to create the organization then becomes the Management Account for the organization. The Management Account used to be called the Master Account. If you hear either of these terms—Management Account or Master Account—just know that they mean the same thing.

      This is a key point to understand with regards to AWS Organizations because the Management Account is special for two reasons, which I’ll explain in this lesson. For now, I’ll add a crown to this account to indicate that it’s the Management Account and to help you distinguish it from other AWS accounts.

      Using this Management Account, you can invite other existing standard AWS accounts into the organization. Since these are existing accounts, they need to approve the invites to join the organization. Once they do, those Standard Accounts will become part of the AWS Organization.

      When standard AWS accounts join an AWS Organization, they change from being Standard Accounts to being Member Accounts of that organization. Organizations have one and only one Management or Master Account and then zero or more Member Accounts.

      You can create a structure of AWS accounts within an organization, which is useful if you have many accounts and need to group them by business units, functions, or even the development stage of an application. The structure within AWS Organizations is hierarchical, forming an inverted tree.

      At the top of this tree is the root container of the organization. This is just a container for AWS accounts at the top of the organizational structure. Don’t confuse this with the Account Root User, which is the admin user of an AWS account. The organizational root is just a container within an AWS Organization, which can contain AWS accounts, including Member Accounts or the Management Account.

      As well as containing accounts, the organizational root can also contain other containers, known as organizational units (OUs). These organizational units can contain AWS accounts, Member Accounts, or the Management Account, or they can contain other organizational units, allowing you to build a complex nested AWS account structure within Organizations.

      Again, please don’t confuse the organizational root with the AWS Account Root User. The AWS Account Root User is specific to each AWS account and provides full permissions over that account. The root of an AWS Organization is simply a container for AWS accounts and organizational units and is the top level of the hierarchical structure within AWS Organizations.

      One important feature of AWS Organizations is consolidated billing. With the example onscreen now, there are four AWS accounts, each with its own billing information. Once these accounts are added to an AWS Organization, the individual billing methods for the Member Accounts are removed. Instead, the Member Accounts pass their billing through to the Management Account of the organization.

      In the context of consolidated billing, you might see the term Payer Account. The Payer Account is the AWS account that contains the payment method for the organization. So, if you see Master Account, Management Account, or Payer Account, know that within AWS Organizations, they all refer to the same thing: the account used to create the organization and the account that contains the payment method for all accounts within the AWS Organization.

      Using consolidated billing within an AWS Organization means you receive a single monthly bill contained within the Management Account. This bill covers the Management Account and all Member Accounts of the organization. One bill contains all the billable usage for all accounts within the AWS Organization, removing a significant amount of financial admin overhead for larger businesses. This alone would be worth creating an organization for most larger enterprises.

      But it gets better. With AWS, certain services become cheaper the more you use them, and for certain services, you can pay in advance for cheaper rates. When using Organizations, these benefits are pooled, allowing the organization to benefit as a whole from the spending of each AWS account within it.

      AWS Organizations also features a service called Service Control Policies (SCPs), which allows you to restrict what AWS accounts within the organization can do. These are important, and I’ll cover them in their own dedicated lesson, which is coming up soon. I wanted to mention them now as a feature of AWS Organizations.

      Before we go through a demo where we'll create an AWS Organization and set up the final account structure for this course, I want to cover two other concepts. You can invite existing accounts into an organization, but you can also create new accounts directly within it. All you need is a valid, unique email address for the new account, and AWS will handle the rest. Creating accounts directly within the organization avoids the invite process required for existing accounts.

      Using an AWS Organization changes what is best practice in terms of user logins and permissions. With Organizations, you don’t need to have IAM Users inside every single AWS account. Instead, IAM roles can be used to allow IAM Users to access other AWS accounts. We’ll implement this in the following demo lesson. Best practice is to have a single account for logging in, which I’ve shown in this diagram as the Management Account of the organization. Larger enterprises might keep the Management Account clean and have a separate account dedicated to handling logins.

      Both approaches are fine, but be aware that the architectural pattern is to have a single AWS account that contains all identities for logging in. Larger enterprises might also have their own existing identity system and may use Identity Federation to access this single identity account. You can either use internal AWS identities with IAM or configure AWS to allow Identity Federation so that your on-premises identities can access this designated login account.

      From there, we can use this account with these identities and utilize a feature called role switching. Role switching allows users to switch roles from this account into other Member Accounts of the organization. This process assumes roles in these other AWS accounts. It can be done from the console UI, hiding much of the technical complexity, but it’s important to understand how it works. Essentially, you either log in directly to this login account using IAM identities or use Identity Federation to gain access to it, and then role switch into other accounts within the organization.

      I’ll discuss this in-depth as we progress through the course. The next lesson is a demo where you’ll implement this yourself and create the final AWS account structure for the remainder of the course.

      Okay, so at this point, it's time for a demo. As I mentioned, you'll be creating the account structure you'll use for the rest of the course. At the start, I demoed creating AWS accounts, including a general AWS account and a production AWS account. In the next lesson, I’ll walk you through creating an AWS Organization using this general account, which will become the Management Account for the AWS Organization. Then, you'll invite the existing production account into the organization, making it a Member Account. Finally, you'll create a new account within the organization, which will be the Development Account.

      I’m excited for this, and it’s going to be both fun and useful for the exam. So, go ahead and finish this video, and when you're ready, I look forward to you joining me in the next lesson, which will be a demo.

    1. Welcome back.

      In this lesson, I want to continue immediately from the last one by discussing when and where you might use IAM roles. By talking through some good scenarios for using roles, I want to make sure that you're comfortable with selecting these types of situations where you would choose to use an IAM role and where you wouldn't, because that's essential for real-world AWS usage and for answering exam questions correctly.

      So let's get started.

      One of the most common uses of roles within the same AWS account is for AWS services themselves. AWS services operate on your behalf and need access rights to perform certain actions. An example of this is AWS Lambda. Now, I know I haven't covered Lambda yet, but it's a function as a service product. What this means is that you give Lambda some code and create a Lambda function. This function, when it runs, might do things like start and stop EC2 instances, perform backups, or run real-time data processing. What it does exactly isn't all that relevant for this lesson. The key thing, though, is that a Lambda function, as with most AWS things, has no permissions by default. A Lambda function is not an AWS identity. It's a component of a service, and so it needs some way of getting permissions to do things when it runs. Running a Lambda function is known as a function invocation or a function execution using Lambda terminology.

      So anything that's not an AWS identity, this might be an application or a script running on a piece of compute hardware somewhere, needs to be given permissions on AWS using access keys. Rather than hard-coding some access keys into your Lambda function, there's actually a better way. To provide these permissions, we can create an IAM role known as a Lambda execution role. This execution role has a trust policy which trusts the Lambda service. This means that Lambda is allowed to assume that role whenever a function is executed. This role has a permissions policy which grants access to AWS products and services.

      When the function runs, it uses the sts:AssumeRole operation, and then the Secure Token Service generates temporary security credentials. These temporary credentials are used by the runtime environment in which the Lambda function runs to access AWS resources based on the permissions the role’s permissions policy has. The code is running in a runtime environment, and it's the runtime environment that assumes the role. The runtime environment gets these temporary security credentials, and then the whole environment, which the code is running inside, can use these credentials to access AWS resources.

      So why would you use a role for this? What makes this scenario perfect for using a role? Well, if we didn't use a role, you would need to hard-code permissions into the Lambda function by explicitly providing access keys for that function to use. Where possible, you should avoid doing that because, A, it's a security risk, and B, it causes problems if you ever need to change or rotate those access keys. It's always better for AWS products and services, where possible, to use a role, because when a role is assumed, it provides a temporary set of credentials with enough time to complete a task, and then these are discarded.

      For a given Lambda function, you might have one copy running at once, zero copies, 50 copies, a hundred copies, or even more. Because you can't determine this number, because it's unknown, if you remember my rule from the previous lesson, if you don't know the number of principals, if it's multiple or if it's an uncertain number, then it suggests a role might be the most ideal identity to use. In this case, it is the ideal way of providing Lambda with these credentials to use a role and allow it to get these temporary credentials. It's always the preferred option when using AWS services to do something on your behalf; use a role because you don't need to provide any static credentials.

      Okay, so let's move on to the next scenario.

      Another situation where roles are useful is emergency or out-of-the-usual situations. Here’s a familiar scenario that you might find in a workplace. This is Wayne, and Wayne works in a business's service desk team. This team is given read-only access to a customer's AWS account so that they can keep an eye on performance. The idea is that anything more risky than this read-only level of access is handled by a more senior technical team. We don't want to give Wayne's team long-term permissions to do anything more destructive than this read-only access, but there are always going to be situations which occur when we least want them, normally 3:00 a.m. on a Sunday morning, when a customer might call with an urgent issue where they need Wayne's help to maybe stop or start an instance, or maybe even terminate an EC2 instance and recreate it.

      So 99% of the time, Wayne and his team are happy with this read-only access, but there are situations when he needs more. This is a break-glass style situation, which is named after this. The idea of break glass in the physical world is that there is a key for something behind glass. It might be a key for a room that a certain team doesn't normally have access to, maybe it’s a safe or a filing cabinet. Whatever it is, the glass provides a barrier, meaning that when people break it, they really mean to break it. It’s a confirmation step. So if you break a piece of glass to get a key to do something, there needs to be an intention behind it. Anyone can break the glass and retrieve the key, but having the glass results in the action only happening when it's really needed. At other times, whatever the key is for remains locked. And you can also tell when it’s been used and when it hasn’t.

      A role can perform the same thing inside an AWS account. Wayne can assume an emergency role when absolutely required. When he does, he'll gain additional permissions based on the role's permissions policy. For a short time, Wayne will, in effect, become the role. This access will be logged and Wayne will know to only use the role under exceptional circumstances. Wayne’s normal permissions can remain at read-only, which protects him and the customer, but he can obtain more if required when it’s really needed. So that’s another situation where a role might be a great solution.

      Another scenario when roles come in handy is when you're adding AWS into an existing corporate environment. You might have an existing physical network and an existing provider of identities, known as an identity provider, that your staff use to log into various systems. For the sake of this example, let’s just say that it's Microsoft Active Directory. In this scenario, you might want to offer your staff single sign-on, known as SSO, allowing them to use their existing logins to access AWS. Or you might have upwards of 5,000 accounts. Remember, there’s the 5,000 IAM user limit. So for a corporation with more than 5,000 staff, you can’t offer each of them an IAM user. That is beyond the capabilities of IAM.

      Roles are often used when you want to reuse your existing identities for use within AWS. Why? Because external accounts can’t be used directly. You can’t access an S3 bucket directly using an Active Directory account. Remember this fact. External accounts or external identities cannot be used directly to access AWS resources. You can’t directly use Facebook, Twitter, or Google identities to interact with AWS. There is a separate process which allows you to use these external identities, which I’ll be talking about later in the course.

      Architecturally, what happens is you allow an IAM role inside your AWS account to be assumed by one of the external identities, which is in Active Directory in this case. When the role is assumed, temporary credentials are generated and these are used to access the resources. There are ways that this is hidden behind the console UI so that it appears seamless, but that's what happens behind the scenes. I'll be covering this in much more detail later in the course when I talk about identity federation, but I wanted to introduce it here because it is one of the major use cases for IAM roles.

      Now, why roles are so important when an existing ID provider such as Active Directory is involved is that, remember, there is this 5,000 IAM user limit in an account. So if your business has more than 5,000 accounts, then you can’t simply create an IAM user for each of those accounts, even if you wanted to. 5,000 is a hard limit. It can't be changed. Even if you could create more than 5,000 IAM users, would you actually want to manage 5,000 extra accounts? Using a role in this way, so giving permissions to an external identity provider and allowing external identities to assume this role, is called ID Federation. It means you have a small number of roles to manage and external identities can use these roles to access your AWS resources.

      Another common situation where you might use roles is if you're designing the architecture for a popular mobile application. Maybe it's a ride-sharing application which has millions of users. The application needs to store and retrieve data from a database product in AWS, such as DynamoDB. Now, I've already explained two very important but related concepts on the previous screen. Firstly, that when you interact with AWS resources, you need to use an AWS identity. And then secondly, that there’s this 5,000 IAM user limit per account. So designing an application with this many users which needs access to AWS resources, if you could only use IAM users or identities in AWS, it would be a problem because of this 5,000 user limit. It’s a hard limit and it can’t be raised.

      Now, this is a problem which can be fixed with a process called Web Identity Federation, which uses IAM roles. Most mobile applications that you’ve used, you might have noticed they allow you to sign in using a web identity. This might be Twitter, Facebook, Google, and potentially many others. If we utilize this architecture for our web application, we can trust these identities and allow these identities to assume an IAM role. This is based on that role’s trust policy. So they can assume that role, gain access to temporary security credentials, and use those credentials to access AWS resources, such as DynamoDB. This is a form of Web Identity Federation, and I'll be covering it in much more detail later in the course.

      The use of roles in this situation has many advantages. First, there are no AWS credentials stored in the application, which makes it a much more preferred option from a security point of view. If an application is exploited for whatever reason, there’s no chance of credentials being leaked, and it uses an IAM role which you can directly control from your AWS account. Secondly, it makes use of existing accounts that your customers probably already have, so they don't need yet another account to access your service. And lastly, it can scale to hundreds of millions of users and beyond. It means you don’t need to worry about the 5,000 user IAM limit. This is really important for the exam. There are very often questions on how you can architect solutions which will work for mobile applications. Using ID Federation, so using IAM roles, is how you can accomplish that. And again, I'll be providing much more information on ID Federation later in the course.

      Now, one scenario I want to cover before we finish up this lesson is cross-account access. In an upcoming lesson, I’ll be introducing AWS Organizations and you will get to see this type of usage in practice. It’s actually how we work in a multi-account environment. Picture the scenario that's on screen now: two AWS accounts, yours and a partner account. Let’s say your partner organization offers an application which processes scientific data and they want you to store any data inside an S3 bucket that’s in their account. Your account has thousands of identities, and the partner IT team doesn’t want to create IAM users in their account for all of your staff. In this situation, the best approach is to use a role in the partner account. Your users can assume that role, get temporary security credentials, and use those to upload objects. Because the IAM role in the partner account is an identity in that account, using that role means that any objects that you upload to that bucket are owned by the partner account. So it’s a very simple way of handling permissions when operating between accounts.

      Roles can be used cross-account to give access to individual resources like S3 in the onscreen example, or you can use roles to give access to a whole account. You’ll see this in the upcoming AWS Organization demo lesson. In that lesson, we’re going to configure it so a role in all of the different AWS accounts that we’ll be using for this course can be assumed from the general account. It means you won’t need to log in to all of these different AWS accounts. It makes multi-account management really simple.

      I hope by this point you start to get a feel for when roles are used. Even if you’re a little vague, you will learn more as you go through the course. For now, just a basic understanding is enough. Roles are difficult to understand at first, so you’re doing well if you’re anything but confused at this point. I promise you, as we go through the course and you get more experience, it will become second nature.

      So at this point, that’s everything I wanted to cover. Thanks for watching. Go ahead and complete this video, and when you're ready, join me in the next lesson.

    1. Welcome back.

      Over the next two lessons, I'll be covering a topic which is usually one of the most difficult identity-related topics in AWS to understand, and that's IAM roles. In this lesson, I'll step through how roles work, their architecture, and how you technically use a role. In the following lesson, I'll compare roles to IAM users and go into a little bit more detail on when you generally use a role, so some good scenarios which fit using an IAM role. My recommendation is that you watch both these lessons back to back in order to fully understand IAM roles.

      So let's get started.

      A role is one type of identity which exists inside an AWS account. The other type, which we've already covered, are IAM users. Remember the term "principal" that I introduced in the previous few lessons? This is a physical person, application, device, or process which wants to authenticate with AWS. We defined authentication as proving to AWS that you are who you say you are. If you authenticate, and if you are authorized, you can then access one or more resources.

      I also previously mentioned that an IAM user is generally designed for situations where a single principal uses that IAM user. I’ve talked about the way that I decide if something should use an IAM user: if I can imagine a single thing—one person or one application—who uses an identity, then generally under most circumstances, I'd select to use an IAM user.

      IAM roles are also identities, but they're used much differently than IAM users. A role is generally best suited to be used by an unknown number or multiple principals, not just one. This might be multiple AWS users inside the same AWS account, or it could be humans, applications, or services inside or outside of your AWS account who make use of that role. If you can't identify the number of principals which use an identity, then it could be a candidate for an IAM role. Or if you have more than 5,000 principals, because of the number limit for IAM users, it could also be a candidate for an IAM role.

      Roles are also something which is generally used on a temporary basis. Something becomes that role for a short period of time and then stops. The role isn't something that represents you. A role is something which represents a level of access inside an AWS account. It's a thing that can be used, short term, by other identities. These identities assume the role for a short time, they become that role, they use the permissions that that role has, and then they stop being that role. It’s not like an IAM user, where you login and it’s a representation of you, long term. With a role, you essentially borrow the permissions for a short period of time.

      I want to make a point of stressing that distinction. If you're an external identity—like a mobile application, maybe—and you assume a role inside my AWS account, then you become that role and you gain access to any access rights that that role has for a short time. You essentially become an identity in my account for a short period of time.

      Now, this is the point where most people get a bit confused, and I was no different when I first learned about roles. What's the difference between logging into a user and assuming a role? In both cases, you get the access rights that that identity has.

      Before we get to the end of this pair of lessons, so this one and the next, I think it's gonna make a little bit more sense, and definitely, as you go through the course and get some practical exposure to roles, I know it's gonna become second nature.

      IAM users can have identity permissions policies attached to them, either inline JSON or via attached managed policies. We know now that these control what permissions the identity gets inside AWS. So whether these policies are inline or managed, they're properly referred to as permissions policies—policies which grant, so allow or deny, permissions to whatever they’re associated with.

      IAM roles have two types of policies which can be attached: the trust policy and the permissions policy. The trust policy controls which identities can assume that role. With the onscreen example, identity A is allowed to assume the role because identity A is allowed in the trust policy. Identity B is denied because that identity is not specified as being allowed to assume the role in the trust policy.

      The trust policy can reference different things. It can reference identities in the same account, so other IAM users, other roles, and even AWS services such as EC2. A trust policy can also reference identities in other AWS accounts. As you'll learn later in the course, it can even allow anonymous usage of that role and other types of identities, such as Facebook, Twitter, and Google.

      If a role gets assumed by something which is allowed to assume it, then AWS generates temporary security credentials and these are made available to the identity which assumed the role. Temporary credentials are very much like access keys, which I covered earlier in the course, but instead of being long-term, they're time-limited. They only work for a certain period of time before they expire. Once they expire, the identity will need to renew them by reassuming the role, and at that point, new credentials are generated and given to the identity again which assumed that role.

      These temporary credentials will be able to access whatever AWS resources are specified within the permissions policy. Every time the temporary credentials are used, the access is checked against this permissions policy. If you change the permissions policy, the permissions of those temporary credentials also change.

      Roles are real identities and, just like IAM users, roles can be referenced within resource policies. So if a role can access an S3 bucket because a resource policy allows it or because the role permissions policy allows it, then anything which successfully assumes the role can also access that resource.

      You’ll get a chance to use roles later in this section when we talk about AWS Organizations. We’re going to take all the AWS accounts that we’ve created so far and join them into a single organization, which is AWS’s multi-account management product. Roles are used within AWS Organizations to allow us to log in to one account in the organization and access different accounts without having to log in again. They become really useful when managing a large number of accounts.

      When you assume a role, temporary credentials are generated by an AWS service called STS, or the Secure Token Service. This is the operation that's used to assume the role and get the credentials, so sts .

      In this lesson, I focused on the technical aspect of roles—mainly how they work. I’ve talked about the trust policy, the permissions policy, and how, when you assume a role, you get temporary security credentials. In the next lesson, I want to step through some example scenarios of where roles are used, and I hope by the end of that, you’re gonna be clearer on when you should and shouldn’t use roles.

      So go ahead, finish up this video, and when you’re ready, you can join me in the next lesson.

    1. Welcome back and welcome to this demo of the functionality provided by IAM Groups.

      What we're going to do in this demo is use the same architecture that we had in the IAM users demo—the SALLI user and those two S3 buckets—but we’re going to migrate the permissions that the SALLI user has from the user to a group that SALLI is a member of.

      Before we get started, just make sure that you are logged in as the IAM admin user of the general AWS account. As always, you’ll need to have the Northern Virginia region selected.

      Attached to this video is a demo files link that will download all of the files you’re going to use throughout the demo. To save some time, go ahead and click on that link and start the file downloading. Once it’s finished, go ahead and extract it; it will create a folder containing all of the files you’ll need as you move through the demo.

      You should have deleted all of the infrastructure that you used in the previous demo lesson. So at this point, we need to go ahead and recreate it. To do that, attached to this lesson is a one-click deployment link. So go ahead and click that link. Everything is pre-populated, so you need to make sure that you put in a suitable password that doesn’t breach any password policy on your account. I’ve included a suitable default password with some substitutions, so that should be okay for all common password policies.

      Scroll down to the bottom, click on the capabilities checkbox, and then create the stack. That’ll take a few moments to create, so I’m going to pause the video and resume it once that stack creation has completed.

      Okay, so that’s created now. Click on Services and open the S3 console in a new tab. This can be a normal tab. Go to the Cat Pics bucket, click Upload, add file, locate the demo files folder that you downloaded and extracted earlier. Inside that folder should be a folder called Cat Pics. Go in there and then select merlin.jpg. Click on Open and Upload. Wait for that to finish.

      Once it’s finished, go back to the console, go to Animal Pics, click Upload again, add files. This time, inside the Animal Pics folder, upload thaw.jpg. Click Upload. Once that’s done, go back to CloudFormation, click on Resources, and click on the Sally user. Inside the Sally user, click on Add Permissions, Attach Policies Directly, select the "Allow all S3 except cats" policy, click on Next, and then Add Permissions.

      So that brings us to the point where we were in the IAM users demo lesson. That’s the infrastructure set back up in exactly the same way as we left the IAM users demo. Now we can click on Dashboard. You’ll need to copy the IAM signing users link for the general account. Copy that into your clipboard.

      You’re going to need a separate browser, ideally, a fully separate browser. Alternatively, you can use a private browsing tab in your current browser, but it’s just easier to understand probably for you at this point in your learning if you have a separate browser window. I’m going to use an isolated tab because it’s easier for me to show you.

      You’ll need to paste in this IAM URL because now we’re going to sign into this account using the Sally user. Go back to CloudFormation, click on Outputs, and you’ll need the Sally username. Copy that into your clipboard. Go back to this separate browser window and paste that in. Then, back to CloudFormation, go to the Parameters tab and get the password for the Sally user. Enter the password that you chose for Sally when you created the stack.

      Then move across to the S3 console and just verify that the Sally user has access to both of these buckets. The easiest way of doing that is to open both of these animal pictures. We’ll start with Thor. Thor’s a big doggo, so it might take some time for him to load in. There we go, he’s loaded in. And the Cat Pics bucket. We get access denied because remember, Sally doesn’t have access to the Cat Pics bucket. That’s as intended.

      Now we’ll go back to our other browser window—the one where we logged into the general account as the IAM admin user. This is where we’re going to make the modifications to the permissions. We’re going to change the permissions over to using a group rather than directly on the Sally user.

      Click on the Resources tab first and select Sally to move across to the Sally user. Note how Sally currently has this managed policy directly attached to her user. Step one is to remove that. So remove this managed policy from Sally. Detach it. This now means that Sally has no permissions on S3. If we go back to the separate browser window where we’ve got Sally logged in and then hit refresh, we see she doesn’t have any permissions now on S3.

      Now back to the other browser, back to the one where we logged in as IAM admin, click on User Groups. We’re going to create a Developers group. Click on Create New Group and call it Developers. That’s the group name. Then, down at the bottom here, this is where we can attach a managed policy to this group. We’re going to attach the same managed policy that Sally had previously directly on her user—Allow all S3 except cats.

      Type "allow" into the filter box and press Enter. Then check the box to select this managed policy. We could also directly at this stage add users to this group, but we’re not going to do that. We’re going to do that as a separate process. So click on 'Create Group'.

      So that’s the Developers group created. Notice how there are not that many steps to create a group, simply because it doesn’t offer that much in the way of functionality. Open up the group. The only options you see here are 'User Membership' and any attached permissions. Now, as with a user, you can attach inline policies or managed policies, and we’ve got the managed policy.

      What we’re going to do next is click on Users and then Add Users to Group. We’re going to select the Sally IAM user and click on Add User. Now our IAM user Sally is a member of the Developers group, and the Developers group has this attached managed policy that allows them to access everything on S3 except the Cat Pics bucket.

      Now if I move back to my other browser window where I’ve got the Sally user logged in and then refresh, now that the Sally user has been added to that group, we’ve got permissions again over S3. If I try to access the Cat Pics bucket, I won’t be able to because that managed policy that the Developers team has doesn’t include access for this. But if I open the Animal Pics bucket and open Thor again—Thor’s a big doggo, so it’ll take a couple of seconds—it will load in that picture absolutely fine.

      So there we go, there’s Thor. That’s pretty much everything I wanted to demonstrate in this lesson. It’s been a nice, quick demo lesson. All we’ve done is create a new group called Developers, added Sally to this Developers group, removed the managed policy giving access to S3 from Sally directly, and added it to the Developers group that she’s now a member of. Note that no matter whether the policy is attached to Sally directly or attached to a group that Sally is a member of, she still gets those permissions.

      That’s everything I wanted to cover in this demo lesson. So before we finish up, let’s just tidy up our account. Go to Developers and then detach this managed policy from the Developers group. Detach it, then go to Groups and delete the Developers group because it wasn’t created as part of the CloudFormation template.

      Then, as the IAM admin user, open up the S3 console. We need to empty both of these buckets. Select Cat Pics, click on Empty. You’ll need to type or copy and paste 'Permanently Delete' into that box and confirm the deletion. Click Exit. Then select the Animal Pics bucket and do the same process. Copy and paste 'Permanently Delete' and confirm by clicking on Empty and then Exit.

      Now that we’ve done that, we should have no problems opening up CloudFormation, selecting the IAM stack, and then hitting Delete. Note if you do have any errors deleting this stack, just go into the stack, select Events, and see what the status reason is for any of those deletion problems. It should be fairly obvious if it can’t delete the stack because it can’t delete one or more resources, and it will give you the reason why.

      That being said, at this point, assume the stack deletions worked successfully, and we’ve cleaned up our account. That’s everything I wanted to cover in this demo lesson. Go ahead, complete this video, and when you’re ready, I’ll see you in the next lesson.

    1. Welcome back.

      In this lesson, I want to briefly cover IAM groups, so let's get started.

      IAM groups, simply put, are containers for IAM users. They exist to make organizing large sets of IAM users easier. You can't log in to IAM groups, and IAM groups have no credentials of their own. The exam might try to trick you on this one, so it's definitely important that you remember you cannot log into a group. If a question or answer suggests logging into a group, it's just simply wrong. IAM groups have no credentials, and you cannot log into them. So they're used solely for organizing IAM users to make management of IAM users easier.

      So let's look at a visual example. We've got an AWS account, and inside it we've got two groups: Developers and QA. In the Developers group, we've got Sally and Mike. In the QA group, we've got Nathalie and Sally. Now, the Sally user—so the Sally in Developers and the Sally in the QA group—that's the same IAM user. An IAM user can be a member of multiple IAM groups. So that's important to remember for the exam.

      Groups give us two main benefits. First, they allow effective administration-style management of users. We can make groups that represent teams, projects, or any other functional groups inside a business and put IAM users into those groups. This helps us organize.

      Now, the second benefit, which builds off the first, is that groups can actually have policies attached to them. This includes both inline policies and managed policies. In the example on the screen now, the Developers group has a policy attached, as does the QA group. There’s also nothing to stop IAM users, who are themselves within groups, from having their own inline or managed policies. This is the case with Sally.

      When an IAM user such as Sally is added as a member of a group—let’s say the Developers group—that user gets the policies attached to that group. Sally gains the permissions of any policies attached to the Developers group and any other groups that that user is a member of. So Sally also gets the policies attached to the QA group, and Sally has any policies that she has directly.

      With this example, Sally is a member of the Developers group, which has one policy attached, a member of the QA group with an additional policy attached, and she has her own policy. AWS merges all of those into a set of permissions. So effectively, she has three policies associated with her user: one directly, and one from each of the group memberships that her user has.

      When you're thinking about the allow or deny permissions in policy statements for users that are in groups, you need to consider those which apply directly to the user and their group memberships. Collect all of the policy allows and denies that a user has directly and from their groups, and apply the same deny-allow-deny rule to them as a collection. Evaluating whether you're allowed or denied access to a resource doesn’t become any more complicated; it’s just that the source of those allows and denies can broaden when you have users that are in multiple IAM groups.

      I mentioned last lesson that an IAM user can be a member of up to 10 groups and there is a 5,000 IAM user limit for an account. Neither of those are changeable; they are hard limits. There’s no effective limit for the number of users in a single IAM group, so you could have all 5,000 IAM users in an account as members of a single IAM group.

      Another common area of trick questions in the exam is around the concept of an all-users group. There isn't actually a built-in all-users group inside IAM, so you don’t have a single group that contains all of the members of that account like you do with some other identity management solutions. In IAM, you could create a group and add all of the users in that account into the group, but you would need to create and manage it yourself. So that doesn’t exist natively.

      Another really important limitation of groups is that you can’t have any nesting. You can’t have groups within groups. IAM groups contain users and IAM groups can have permissions attached. That’s it. There’s no nesting, and groups cannot be logged into; they don’t have any credentials.

      Now, there is a limit of 300 groups per account, but this can be increased with a support ticket.

      There’s also one more point that I want to make at this early stage in the course. This is something that many other courses tend to introduce later on or at a professional level, but it's important that you understand this from the very start. I'll show you later in the course how policies can be attached to resources, for example, S3 buckets. These policies, known as resource policies, can reference identities. For example, a bucket could have a policy associated with it that allows Sally access to that bucket. That’s a resource policy. It controls access to a specific resource and allows or denies identities to access that bucket.

      It does this by referencing these identities using an ARN, or Amazon Resource Name. Users and IAM roles, which I'll be talking about later in the course, can be referenced in this way. So a policy on a resource can reference IAM users and IAM roles by using the ARN. A bucket could give access to one or more users or to one or more roles, but groups are not a true identity. They can’t be referenced as a principal in a policy. A resource policy cannot grant access to an IAM group. You can grant access to IAM users, and those users can be in groups, but a resource policy cannot grant access to an IAM group. It can’t be referred to in this way. You couldn’t have a resource policy on an S3 bucket and grant access to the Developers group and then expect all of the developers to access it. That’s not how groups work. Groups are just there to group up IAM users and allow permissions to be assigned to those groups, which the IAM users inherit.

      So this is an important one to remember, whether you are answering an exam question that involves groups, users, and roles or resource policies, or whether you're implementing real-world solutions. It’s easy to overestimate the features that a group provides. Don’t fall into the trap of thinking that a group offers more functionality than it does. It’s simply a container for IAM users. That’s all it’s for. It can contain IAM users and have permissions associated with it; that’s it. You can’t log in to them and you can’t reference them from resource policies.

      Okay, so that’s everything I wanted to cover in this lesson. Go ahead, complete the video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back.

      In this demo lesson, we're going to explore IAM users. This is the first type of identity that we've covered in AWS. We'll use the knowledge gained from the IAM policy documents lesson to assign permissions to an IAM user in our AWS account.

      To get started, you'll need to be logged in as the IAM admin user to the general AWS account and have the Northern Virginia region selected.

      Attached to this lesson are two links. The first is a one-click deployment link, which will deploy the infrastructure needed for this demo. The second link will download the files required for this demo. Click the demo files link to start the download, and then click the one-click deployment link to begin the deployment.

      Earlier in the course, you created an IAM user that you should be logged into this account with. I won't go through the process of creating an IAM user again. Instead, we'll use CloudFormation to apply a template that will create an IAM user named Sally, along with two S3 buckets for this demonstration and a managed policy. Enter a password for the Sally user as a parameter in the CloudFormation stack. Use a password that’s reasonably secure but memorable and typeable. This password must meet the password policy assigned to your AWS account, which typically requires a minimum length of eight characters and a mix of character types, including uppercase, lowercase, numbers, and certain special characters. It also cannot be identical to your AWS account name or email address.

      After entering the password, scroll down, check the capabilities box, and click on create stack. Once the stack is created, switch to your text editor to review what the template does. It asks for a parameter (Sally's password) and creates several resources.

      There are logical resources called 'catpix' and 'animalpix,' both of which are S3 buckets. Another logical resource is 'Sally,' an IAM user. This IAM user has a managed policy attached to it, which references an ARN for a managed policy that will be shown once the stack is complete. It also sets the login profile, including the password, and requires a password reset upon first login.

      The managed policy created allows access to S3 but denies access to the CatPix S3 bucket. This setup is defined in the policy logical resource, which you’ll see once the stack is complete.

      Returning to the AWS console, refresh the page. You should see the four created resources: the AnimalPix S3 bucket, the CatPix S3 bucket, the IAM Managed Policy, and the Sally IAM User.

      CloudFormation generates resource names by taking the stack name ("iam"), the logical resource name defined in the template ("animal_pics," "cat_pics," and "sally"), and adding some randomness to ensure unique physical IDs.

      Now, open the Sally IAM user by clicking the link under Resources. Note that Sally has an attached managed policy called IAM User Change Password, which allows her to change her password upon first login.

      Go to Policies in the IAM console to see the managed policies inside the account. The IAM User Change Password policy is one of the AWS managed policies and allows Sally to change her password.

      Next, click on Dashboard and ensure that you have the IAM users sign-in link on your clipboard. Open a private browser tab or a separate browser to avoid logging out of your IAM admin user. Use this separate tab to access the IAM sign-in page for the general AWS account.

      Retrieve Sally’s username from CloudFormation by clicking on outputs and copying it. Paste this username into the IAM username box on the sign-in page, and enter the password you chose for Sally. Click on sign in.

      After logging in as Sally, you’ll need to change the password. Enter the old password, then choose and confirm a new secure password. This is possible due to the managed policy assigned to Sally that allows her to change her password.

      Once logged in, test Sally’s permissions by navigating to the EC2 console. You might encounter API errors or lack of permissions. Check the S3 console and note that you won't have permissions to list any S3 buckets, even though we know at least two were created. This demonstrates that IAM users initially have no permissions apart from changing their passwords.

      Locate and extract the zip file downloaded from the demo files link. Inside the extracted folder, open the file named S3_FullAdminJSON. This JSON policy document grants full access to any S3 actions on any S3 resource.

      Assign this as an inline policy to Sally by copying the JSON policy document and pasting it into the IAM console. Go to the IAM area, open the Sally user, go to the Permissions tab, and click Add Permissions, then Create Inline Policy. Select the JSON tab, delete any existing document, and paste in the JSON policy.

      Review the policy, name it S3 Admin Inline, and click Create Policy. Sally will now have this S3 Admin Inline policy in addition to the IAM User Change Password managed policy.

      Switch to the browser or tab logged in as Sally and refresh the page. You should now be able to see the S3 buckets. Upload a file to both the AnimalPix and CatPix buckets to verify permissions. For example, upload thor.jpg to AnimalPix and merlin.jpg to CatPix.

      To ensure you can read from the CatPix bucket, click on merlin.jpg and select open. You should see the file, confirming that you have access.

      Return to the browser logged in as the IAM admin user. Open the Sally user and delete the S3 Admin Inline policy. This will remove her access rights over S3.

      In the other browser or tab logged in as Sally, refresh the page. You should see an access denied error for S3 actions on the CatPix bucket, while still having access to the AnimalPix bucket.

      Finally, return to the IAM admin browser or tab. Click on Add Permissions for Sally and attach the managed policy created by the CloudFormation template, "allow all S3 except cats." This policy has two statements: one allowing all S3 actions and another explicitly denying access to the CatPix bucket.

      Verify this by refreshing the page logged in as Sally. You should be able to interact with all S3 buckets except the CatPix bucket.

      To conclude, this demo showed how to apply different types of policies to an IAM user, including inline and managed policies. We demonstrated how these policies affect effective permissions.

      For cleanup, delete the managed policy attachment from Sally. In the S3 console, empty both the CatPix and AnimalPix buckets by typing "permanently delete" and clicking empty. Return to CloudFormation, select the IAM stack, and hit delete to clean up all resources created by this stack.

      That covers everything for this demo. Complete this video, and when you're ready, join me in the next lesson.

    1. Welcome back.

      And in this lesson, I want to finish my coverage of IAM users.

      You already gained some exposure to IAM users earlier in the course. Remember, you created an IAM admin user in both your general and production AWS accounts. As well as creating these users, you secured them using MFA, and you attached an AWS managed policy to give this IAM user admin rights in both of those accounts.

      So for now, I just want to build upon your knowledge of IAM users by adding some extra detail that you'll need for the exam. So let's get started.

      Now, before I go into more detail, let's just establish a foundation. Let's use a definition. Simply put, IAM users are an identity used for anything requiring long-term AWS access. For example, humans, applications, or service accounts. If you need to give something access to your AWS account, and if you can picture one thing, one person or one application—so James from accounts, Mike from architecture, or Miles from development—99% of the time you would use an IAM user.

      If you need to give an application access to your AWS account, for example, a backup application running on people's laptops, then each laptop generally would use an IAM user. If you have a need for a service account, generally a service account which needs to access AWS, then generally this will use an IAM user. If you can picture one thing, a named thing, then 99% of the time, the correct identity to select is an IAM user. And remember this because it will help in the exam.

      IAM starts with a principal. And this is a word which represents an entity trying to access an AWS account. At this point, it's unidentified. Principals can be individual people, computers, services, or a group of any of those things. For a principal to be able to do anything, it needs to authenticate and be authorized. And that's the process that I want to step through now.

      A principal, which in this example, is a person or an application, makes requests to IAM to interact with resources. Now, to be able to interact with resources, it needs to authenticate against an identity within IAM. An IAM user is an identity which can be used in this way.

      Authentication is this first step. Authentication is a process where the principal on the left proves to IAM that it is an identity that it claims to be. So an example of this is that the principal on the left might claim to be Sally, and before it can use AWS, it needs to prove that it is indeed Sally. And it does this by authenticating.

      Authentication for IAM users is done either using username and password or access keys. These are both examples of long-term credentials. Generally, username and passwords are used if a human is accessing AWS and accessing via the console UI. Access keys are used if it's an application, or as you experienced earlier in the course, if it's a human attempting to use the AWS Command Line tools.

      Now, once a principal goes through the authentication process, the principal is now known as an authenticated identity. An authenticated identity has been able to prove to AWS that it is indeed the identity that it claims to be. So it needs to be able to prove that it's Sally. And to prove that it's Sally, it needs to provide Sally's username and password, or be able to use Sally's secret access key, which is a component of the access key set. If it can do that, then AWS will know that it is the identity that it claims to be, and so it can start interacting with AWS.

      Once the principal becomes an authenticated identity, then AWS knows which policies apply to the identity. So in the previous lesson, I talked about policy documents, how they could have one or more statements, and if an identity attempted to access AWS resources, then AWS would know which statements apply to that identity. That's the process of authorization.

      So once a principal becomes an authenticated identity, and once that authenticated identity tries to upload to an S3 bucket or terminate an EC2 instance, then AWS checks that that identity is authorized to do so. And that's the process of authorization. So they're two very distinct things. Authentication is how a principal can prove to IAM that it is the identity that it claims to be using username and password or access keys, and authorization is IAM checking the statements that apply to that identity and either allowing or denying that access.

      Okay, let's move on to the next thing that I want to talk about, which is Amazon Resource Names, or ARNs. ARNs do one thing, and that's to uniquely identify resources within any AWS accounts. When you're working with resources, using the command line or APIs, you need a way to refer to these resources in an unambiguous way. ARNs allow you to refer to a single resource, if needed, or in some cases, a group of resources using wild cards.

      Now, this is required because things can be named in a similar way. You might have an EC2 instance in your account with similar characteristics to one in my account, or you might have two instances in your account but in different regions with similar characteristics. ARNs can always identify single resources, whether they're individual resources in the same account or in different accounts.

      Now, ARNs are used in IAM policies which are generally attached to identities, such as IAM users, and they have a defined format. Now, there are some slight differences depending on the service, but as you go through this course, you'll gain enough exposure to be able to confidently answer any exam questions that involve ARNs. So don't worry about memorizing the format at this stage, you will gain plenty of experience as we go.

      These are two similar, yet very different ARNs. They both look to identify something related to the catgifs bucket. They specify the S3 service. They don't need to specify a region or an account because the naming of S3 is globally unique. If I use a bucket name, then nobody else can use that bucket name in any account worldwide.

      The difference between these two ARNs is the forward slash star on the end at the second one. And this difference is one of the most common ways mistakes can be made inside policies. It trips up almost all architects or admins at one point or another. The top ARN references an actual bucket. If you wanted to allow or deny access to a bucket or any actions on that bucket, then you would use this ARN which refers to the bucket itself. But a bucket and objects in that bucket are not the same thing.

      This ARN references anything in that bucket, but not the bucket itself. So by specifying forward slash star, that's a wild card that matches any keys in that bucket, so any object names in that bucket. This is really important. These two ARNs don't overlap. The top one refers to just the bucket and not the objects in the bucket. The bottom one refers to the objects in the bucket but not the bucket itself.

      Now, some actions that you want to allow or deny in a policy operate at a bucket level or actually create buckets. And this would need something like the top ARN. Some actions work on objects, so it needs something similar to the bottom ARN. And you need to make sure that you use the right one. In some cases, creating a policy that allows a set of actions will need both. If you want to allow access to create a bucket and interact with objects in that bucket, then you would potentially need both of these ARNs in a policy.

      ARNs are collections of fields split by a colon. And if you see a double colon, it means that nothing is between it. It doesn't need to be specified. So in this example, you'll see a number of double colons because you don't need to specify the region or account number for an S3 bucket because the bucket name is globally unique. A star can also be used, which is a wild card.

      Now, keep in mind they're not the same thing. So not specifying a region and specifying star don't mean the same thing. You might use a star when you want to refer to all regions inside an AWS account. Maybe you want to give permissions to interact with EC2 in all regions, but you can't simply omit this. The only place you'll generally use the double colon is when something doesn't need to be specified, you'd use a star when you want to refer to a wild card collection of a set of things. So they're not the same thing. Keep that in mind, and I'll give you plenty of examples as we go through the course.

      So the first field is the partition, and this is the partition that the resource is in. For standard AWS regions, the partition is AWS. If you have resources in other partitions, the partition is AWS-hyphen-partition name. This is almost never anything but AWS. But for example, if you do have resources in the China Beijing region, then this is AWS-cn.

      The next part is service. And this is the service name space that identifies the AWS product. For example, S3, IAM, or RDS. The next field is region. So this is the region that the resource you're referring to resides in. Some ARNs do not require a region, so this might be omitted, and certain ARNs require wild card. And you'll gain exposure through the course as to what different services require for their ARNs.

      The next field is the account ID. This is the account ID of the AWS account that owns the resource. So for example, 123456789012. So if you're referring to an EC2 instance in a certain account, you will have to specify the account number inside the ARN. Some resources don't require that, so this example is S3 because it is globally unique across every AWS account. You don't need to specify the account number.

      And then at the end, we've got resource or resource type. And the content of this part of the ARN varies depending on the service. A resource identifier can be the name or ID of an object. For example, user forward slash Sally or instance forward slash and then the instance ID, or it can be a resource path. But again, I'm only introducing this at this point. You'll get plenty of exposure as you go through the course. I just want to give you this advanced knowledge so you know what to expect.

      So let's quickly talk about an exam PowerUp. I tend not to include useless facts and figures in my course, but some of them are important. This is one such occasion.

      Now first, you can only ever have 5,000 IAM users in a single account. IAM is a global service, so this is a per account limit, not per region. And second, an IAM user can be a member of 10 IAM groups. So that's a maximum. Now, both of these have design impacts. You need to be aware of that.

      What it means is that if you have a system which requires more than 5,000 identities, then you can't use one IAM user for each identity. So this might be a limit for internet scale applications with millions of users, or it might be a limit for large organizations which have more than 5,000 staff, or it might be a limit when large organizations are merging together. If you have any scenario or a project with more than 5,000 identifiable users, so identities, then it's likely that IAM users are not the right identity to pick for that solution.

      Now, there are solutions which fix this. We can use IAM roles or Identity Federation, and I'll be talking about both of those later in the course. But in summary, it means using your own existing identities rather than using IAM users. And I'll be covering the architecture and the implementation of this later in the course.

      At this stage, I want you to take away one key fact, and that is this 5,000 user limit. If you are faced with an exam question which mentions more than 5,000 users, or talks about an application that's used on the internet which could have millions of users, and if you see an answer saying create an IAM user for every user of that application, that is the wrong answer. Generally with internet scale applications, or enterprise access or company mergers, you'll be using Federation or IAM roles. And I'll be talking about all of that later in the course.

      Okay, so that's everything I wanted to cover in this lesson. So go ahead, complete the video, and when you're ready, I'll look forward to you joining me in the next.

    1. Welcome back. In this lesson, I want to start by covering an important aspect of how AWS handles security, specifically focusing on IAM policies.

      IAM policies are a type of policy that gets attached to identities within AWS. As you've previously learned, identities include IAM users, IAM groups, and IAM roles. You’ll use IAM policies frequently, so it’s important to understand them for the exam and for designing and implementing solutions in AWS.

      Policies, once you understand them, are actually quite simple. I’ll walk you through the components and give you an opportunity to experiment with them in your own AWS account. Understanding policies involves three main stages: first, understanding their architecture and how they work; second, gaining the ability to read and understand the policy; and finally, learning to write your own. For the exam, understanding their architecture and being able to read them is sufficient. Writing policies will come as you work through the course and gain more practical experience.

      Let's jump in. An IAM identity policy, or IAM policy, is essentially a set of security statements for AWS. It grants or denies access to AWS products and features for any identity using that policy. Identity policies, also known as policy documents, are created using JSON. Familiarity with JSON is helpful, but if you're new to it, don’t worry—it just requires a bit more effort to learn.

      This is an example of an identity policy document that you would use with a user, group, or role. At a high level, a policy document consists of one or more statements. Each statement is enclosed in curly braces and grants or denies permissions to AWS services.

      When an identity attempts to access AWS resources, it must prove its identity through a process known as authentication. Once authenticated, AWS knows which policies apply to that identity, and each policy can contain multiple statements. AWS also knows which resources you’re trying to interact with and what actions you want to perform on those resources. AWS reviews all relevant statements one by one to determine the permissions for a given identity accessing a particular resource.

      A statement consists of several parts. The first part is a statement ID, or SID, which is optional but helps identify the statement and its purpose. For example, "full access" or "DenyCatBucket" indicates what the statement does. Using these identifiers is considered best practice.

      Every interaction with AWS involves two main elements: the resource and the actions attempted on that resource. For instance, if you’re interacting with an S3 bucket and trying to add an object, the statement will only apply if it matches both the action and the resource. The action part of a statement specifies one or more actions, which can be very specific or use wildcards (e.g., s3:* for all S3 operations). Similarly, resources can be specified individually or in lists, and wildcards can refer to all resources.

      The final component is the effect, which is either "allow" or "deny." The effect determines what AWS does if the action and resource parts of the statement match the attempted operation. If the effect is "allow," access is granted; if it’s "deny," access is blocked. An explicit deny always takes precedence over an explicit allow. If neither applies, the default implicit deny prevails.

      In scenarios where there are multiple policies or statements, AWS evaluates all applicable statements. If there’s an explicit deny, it overrides any explicit allows. If no explicit deny is present, an explicit allow will grant access, unless there’s an explicit deny.

      Lastly, there are two main types of policies: inline policies and managed policies. Inline policies are directly attached to individual identities, making them isolated and cumbersome to manage for large numbers of users. Managed policies are created as separate objects and can be attached to multiple identities, making them more efficient and easier to manage. AWS provides managed policies, but you can also create and manage customer managed policies tailored to your specific needs.

      Before concluding, you’ll have a chance to gain practical experience with these policies. For now, this introduction should give you a solid foundation. Complete the video, and I look forward to seeing you in the next lesson.

    1. Welcome back.

      This is part two of this lesson. We're gonna continue immediately from the end of part one, so let's get started.

      Right at the offset, it's critical to understand that layer 2 of the networking stack uses layer 1. So in order to have an active layer 2 network, you need the layer 1 or physical layer in place and working, meaning two network interfaces and a shared medium between those as a minimum. But now on top of this, each networking stack, so remember this is all of the networking software layers, each stack, so the left and the right, now have both layer 1 and layer 2. So conceptually, layer 2 sits on top of layer 1 like this, and our games, these are now communicating using layer 2.

      Each device, because it's layer 2, now has a MAC address, a hardware address, which is owned by the network interface card on each device. So let's step through what happens now when the game on the left wants to send something to the game running on the right. Well, the left game knows the MAC address of the game on the right. Let's assume that as a starting point. So it communicates with the layer 2 software. Let's assume this is ethernet on the left. It indicates that it wants to send data to the MAC address on the right and the layer 2 software based on this, creates an ethernet frame containing the data that the game wants to send in the payload part of the frame.

      So the frame F1 has a destination of the MAC address ending 5b76, which is the MAC address on the laptop on the right and it contains within the payload part of that frame the data that the game wants to send. At this point, this is where the benefits of layer 2 begin. Layer 2 can communicate with the layer 1 part of the networking stack and it can look for any signs of a carrier signal. If any other device on the network were transmitting at this point, you would see the signal on the layer 1 network. So it's looking to sense a carrier, and this is the job of CSMA, which stands for carrier sense multiple access. In this case, it doesn't detect a carrier and so it passes the frame to layer 1. Layer 1 doesn't care what the frame is, it doesn't understand the frame as anything beyond a block of data and so it transmits this block of data onto the shared medium.

      On the right side, the layer 1 software receives the raw bit stream and it passes it up to its layer 2. Layer 2 reviews the destination MAC address of the frame. It sees that the frame is destined for itself and so it can pass that payload, that data, back to the game and so that's how these games can communicate using layer 2. So layer 2 is using layer one to transmit and receive the raw data, but it's adding on top of this MAC addresses which allow for machine-to-machine communication and in addition, it's adding this media access control.

      So let's look at an example of how this works. If we assume that at the same time that the left machine was transmitting, the machine on the right attempted to do the same. So layer 2 works with layer 1 and it checks for a carrier on the shared physical medium. If the left machine is transmitting, which we know it is, the carrier is detected and layer 2 on the right simply waits until the carrier is not detected. So it's layer 2 which is adding this control. Layer 1 on its own would simply transmit and cause a collision, but layer 2 checks for any carrier before it instructs layer 1 to transmit. When the carrier isn't detected anymore, then layer 2 sends the frame down to layer 1 for transmission. Layer 1 just sees it as data to send and it transmits it across the physical medium. It's received at the left side device. Layer 1 sends this raw data to layer 2. It can see that it's the intended destination and so it sends the data contained in the frame payload back to the game.

      Now, I want to reiterate a term again which you need to remember, and that term is encapsulation. This is the process of taking some data, in this case game data, and wrapping it in something else. In this case, the game data is encapsulated inside a frame at each side before giving the game back its intended data, the data is de-encapsulated, the payload is extracted from the frame. So this is a concept that you need to remember, because as data passes down the OSI model, it's encapsulated in more and more different components. So the transport layer does some encapsulation, the network layer does some encapsulation and the data link layer does some encapsulation. This is a process which you need to be comfortable with.

      Now, two more things which I want to cover on this screen and the first of this is conceptually, for anything using layer 2 to communicate, they see it as layer 2 on the left is directly communicating to layer 2 on the right. Even though layer 2 is using layer 1 to perform the physical communication, anything which is using these layer 2 services has no visibility of that. That's something that's common in the OSI model. Anything below the point that you are communicating with is abstracted away. If you're using a web browser which functions at layer 7, you don't have to worry about how your data gets to the web server. It just works. So your web browser, which is running at layer 7 is communicating with a web server which is also running at layer 7. You don't have to worry about how this communication happens.

      The other thing which I want to cover, is if in this scenario, what if both machines check for a carrier which doesn't exist and then both layer 2s instruct their layer 1 to both transmit at the same time. This causes a collision. Now, layer 2 contains collision detection and that's what the CD part of CSMA/CD is for. If a collision is detected, then a jam signal is sent by all of the devices which detect it and then a random back-off occurs. The back-off is a period of time during which no device will attempt a transmission. So after this back-off period occurs, the transmission is retried. Now, because this back-off is random, hopefully it means that only one device will attempt to transmit at first and other devices will see the carrier on the network and wait before transmitting, but if we still have a collision, then this back-off is attempted again only now with a greater period. So over time, there's less and less likelihood that you're going to have multiple devices transmitting at the same time. So this collision detection and avoidance is essential for layer 2. It's the thing that allows multiple devices to coexist on the same layer 2 network.

      Okay, so let's move on.

      So now you have an idea about layer 2 networking, let's revisit how our laptops are connected. In the previous example where I showed hubs, we had four devices connected to the same four port hub. Now, hubs are layer 1 devices. This means they don't understand frames in any way. They just see physical data. Essentially, they're a multi-port repeater. They just repeat any physical activity on one port to all or the ports. So the top laptop sends a frame destined for the bottom laptop. The hub just sees this as raw data. It repeats this on all of the other ports and this means all of the other laptops will receive this data, which their layer 2 software will interpret as a frame. They'll all receive this data, so this frame, they'll see that they're not the intended destination and they'll discard it. The bottom laptop will see that it is the intended destination and its layer 2 software will pass on the data to the game.

      Now, hubs aren't smart and this means that if the laptop on the right were to start transmitting at exactly the same time, then it would cause a collision, and this collision would be repeated on all of the other ports that the hub has connected. Using a layer 1 device, a hub doesn't prevent you running a layer 2 network over the top of it, but it means that the hub doesn't understand layer 2 and so it behaves in a layer 1 way. You still have each device doing carrier sense multiple access, and so collisions should be reduced, but if multiple devices try and transmit at exactly the same time, then this will still cause collisions.

      What will improve this is using a switch, and a switch is a layer 2 device. It works in the same way physically as a hub, but it understands layer 2 and so it provides significant advantages. Let's review how this changes things. To keep things simple, let's keep the same design only now we have a switch in the middle. It still has four ports and it's still connected to the same four laptops. Because they're layer 2, they now have their own hardware addresses, their MAC addresses. Now, because a switch is a layer 2 device, it means that it has layer 2 software running inside it, which means that it understands layer 2. And because of that it maintains what's called a MAC address table. Switches over time learn what's connected to each port, so the device MAC addresses which are connected to each port. When a switch sees frames, it can interpret them and see the source and destination MAC addresses. So over time with this network, the MAC address table will get populated with each of our devices. So the switch will store the MAC addresses it sees on a port and the port itself. This is generally going to happen the first time each of the laptop sends a frame which the switch receives. It will see the source MAC address on the frame and it will update the MAC address table which it maintains.

      Now let's say the MAC address table is populated and the top laptop sends a frame which is intended for the left laptop. Well, the switch will see the frame arrive at the port that the top laptop is connected to at which point, one of two things will happen. If the switch didn't know which port the destination MAC address was on, well it would forward this frame to all of the other ports. If it does know which port the specific MAC address is attached to, then it will use that one port to forward the frame to. Switches are intelligent. They aren't just repeating the physical level. They interpret the frames and make decisions based on the source and destination MAC addresses of those frames. So switches store and forward frames. They receive the frame, they store it and then they forward it based on the MAC address table and then they discard it. Now this has another benefit, because it's not just repeating like a dumb layer 1 device, it means that it won't forward collisions. In fact, each port on the switch is a separate collision domain. Because each port is a separate collision domain, the only two things which can transmit at the same time are the device and the port it's connected to. So if there is a collision, it will be limited to that one port only. The switch will not forward that corrupted data through to any of its other ports, because it only forwards valid frames. The switch isn't forwarding the physical layer, it's dealing with frames only. It receives a frame. If it's valid, it stores it, it reviews it and then it forwards it.

      Now, layer 2 is the foundation for all networks which you use day-to-day. It's how your wired networks work. It's how wifi networks work. It's how the internet works, which is basically a huge collection of interconnected layer 2 networks. The name itself, so the internet, stands for an inter-network of networks. So these networks are layer 2 networks which are all connected together to form the internet.

      So in summary, what position are we in by adding layer 2?

      Well, we have identifiable devices. For the first time, we can uniquely address frames to a particular device using MAC addresses. It allows for device to device communication rather than the shared media which layer 1 offers. We also have media access control so devices can share access to physical media in a nice way avoiding crosstalk and collisions. But we also have the ability to detect collisions and so if they do occur, we have a way to correct or work around them. So with all of that, we can do unicast communications which are one-to-one communications and we can do broadcast communications, which are one-to-all communications. And as long as we replace layer 1 hubs with layer two switches, which are like hubs with superpowers, then we gain the ability to scale to a much better degree and we avoid a lot of the collisions because switches store and forward rather than just repeating everything.

      Now, layer 2 really does provide crucial functionality. Everything from this point onwards builds on layer 2, so it's critical that you understand it. So if necessary, then re-watch this video, because from now on, you need to understand layer 2 at a real fundamental level. Now, this seems like a great point to take a break, so go ahead and complete this video and when you're ready, join me in the next part of this series where we'll be looking at layer 3, which is the network layer.

    1. Welcome back.

      And in this part of the lesson series, we're going to look at layer two of the OSI model, which is the data link layer.

      Now the data link layer is one of the most critical layers in the entire OSI seven-layer model. Everything above this point relies on the device-to-device communication, which the data link layer provides. So when you are sending or receiving data, to or from the internet, just be aware that the data link layer is supporting the transfer of that data. So it's essential that you understand.

      Now, this is going to be one of the longer parts of this lesson series, because layer two actually provides a significant amount of functionality.

      Now, before I step through the architecture of layer two, we have to start with the fundamentals. Layer two, which is the data link layer, runs over layer one. So a layer two network requires a functional layer one network to operate, and that's something which is common throughout the OSI model. Higher layers build on lower layers adding features and capabilities. A layer two network can run on different types of layer one networks, so copper, fiber, wifi, and provide the same capabilities.

      Now there are different layer two protocols and standards for different situations, but for now, we're going to focus on ethernet which is what most local networks use. So things in your office or things in your home.

      Now, layer two, rather than being focused on physical wavelengths or voltages, introduces the concept of frames. And frames are a format for sending information over a layer two network. Layer two also introduces a unique hardware address, called a MAC address, for every device on a network. This hardware address is a hexadecimal address. It's 48 bits long and it looks like this: 3e:22 and so on. The important thing to understand is that a MAC address, generally, for physical networking, is not software assigned. The address is uniquely attached to a specific piece of hardware.

      A MAC address is formed of two parts, the OUI, which is the organizationally unique identifier, and this is assigned to companies who manufacture network devices. So each of these companies will have a separate OUI. The second part of the MAC address is then network interface controller or NIC specific. And this means together the MAC address on a network card should be globally unique.

      Now, layer two, as I've mentioned, uses layer one. This means that a layer two, or ethernet frame, can be transmitted onto the shared physical medium by layer one. This means that it's converted into voltages, RF, or light. It's sent across the medium and then received by other devices, also connected to that shared medium.

      It's important to understand this distinction. Layer two provides frames, as well as other things which I'll cover soon. And layer one handles the physical transmission and reception onto and from the physical shared medium. So when layer one is transmitting a frame onto the physical medium, layer one doesn't understand the frame. Layer one is simply transmitting raw data onto that physical medium.

      Now a frame, which is the thing that layer two uses for communication, is a container of sorts. It has a few different components. The first part is the preamble and start frame delimiter. And the function of this is to allow devices to know that it's the start of the frame, so they can identify the various parts of that frame. You need to know where the start of a frame is to know where the various parts of that frame start.

      Next comes the destination and the source MAC addresses. So all devices on a layer two network have a unique MAC address, and a frame can be sent to a specific device on a network by putting its MAC address in the destination field, or you can put all Fs if you want to send the frame to every device on the local network. And this is known as a broadcast. Now the source MAC address field is set to the device address of whatever is transmitting the frame and this allows it to receive replies.

      Next is EtherType, and this is commonly used to specify which layer three protocol is putting its data inside a frame. Just like layer two uses layer one to move raw bitstream data across the shared physical medium, while layer three uses layer two frames for device-to-device communication on a local network. And so when you are receiving a frame at the other side of a communication, you need to know which layer three protocol originally put data into that frame. A common example might be IP or the internet protocol. So this is what the EtherType, or ET, field is used for.

      Now, these three fields are commonly known as the MAC header. They control the frame destination, they indicate the source and specify its function. After the header is the payload, and this is anywhere from 46 to 1500 bytes in size for standard frames. It contains the data that the frame is sending. The data is generally provided by the layer three protocol and the protocol which is being used, as I just mentioned, is indicated within the EtherType or ET field.

      Now this process is called encapsulation. You have something which layer three generates, often this is an IP Packet, and this is put inside an ethernet frame. It's encapsulated in that frame. The frame delivers that data to a different layer two destination. On the other side, the frame is analyzed, and the layer three packet is extracted and given back to layer three at the destination side. The EtherType field is used to determine which layer three protocol receives this at the destination. And then finally at the end of the frame, is the frame check sequence, which is used to identify any errors in the frame. It's a simple CRC check. It allows the destination to check if corruption has occurred or not.

      So that's the frame and it's an important thing to understand if you are to understand how all of the bits of layer one, two, and three fit together. So layer two frames are generated by the layer two software at the source side. They're passed to layer one. That raw data is transmitted onto the physical medium. It's taken off the physical medium at the destination side. It's passed to its layer two software and that can interpret the frame and pass that onto layer three, which can then interpret that data.

      Now as a reminder, this is the problem that we have with a purely layer one network implementation. We have two devices running a game, a laptop on the left and a laptop on the right. And these are connected using a single network cable, a shared physical medium. Now, as I mentioned earlier in this lesson series, layer one provides no media access control. The layer one software rolling on the network card will simply transmit any data it receives onto the physical medium. So if the game on the left sends some data, it will be transmitted onto the medium and it will be seen by the device on the right. The problem is that the laptop on the right could also be sending data at the same time. This means the electrical signals will overlap and interfere with each other and this is known as a collision and it impacts both pieces of data. It corrupts both.

      This is one of the problems of layer one, which is solved by layer two, and layer two provides controlled access to the physical medium. Now let's explore how.

      Okay, so this is the end of part one of this lesson. It was getting a little bit on the long side and so I wanted to add a break. It's an opportunity just to take a rest or grab a coffee. Part two will be continuing immediately from the end of part one. So go ahead, complete the video, and when you're ready, join me in part two.

    1. Welcome to this lesson where I'm going to be talking about high availability (HA), fault tolerance (FT), and disaster recovery (DR). It's essential that you understand all three of these to be an effective solutions architect and I want to make sure that you understand all of them correctly. Many of the best architects and consultants that I've worked with have misunderstood exactly what HA and FT mean. The best outcome of this misunderstanding is that you waste business funds and put a project at risk. Worst case, you can literally put lives at risk. So, let's jump in and get started and I promise to keep it as brief as possible, but this really is something you need to fully understand.

      Let's start with high availability. This is a term that most people think that they understand. Formally, the definition is that high availability aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period and I've highlighted the key parts of that definition. Most students that I initially teach have an assumption that making a system highly available means ensuring that the system never fails or that the user of a system never experiences any outages and that is not true. HA isn't aiming to stop failure, and it definitely doesn't mean that customers won't experience outages. A highly available system is one designed to be online and providing services as often as possible. It's a system designed so that when it fails, its components can be replaced or fixed as quickly as possible, often using automation to bring systems back into service. High availability is not about the user experience. If a system fails and a component is replaced and that disrupts service for a few seconds, that's okay. It's still highly available. High availability is about maximizing a system's online time and that's it.

      Let me give you an example. Let's say we have a system which has a customer, Winnie. Winnie is a data scientist and uses a bespoke application to identify complex data trends. Now, this application runs on a single server, let's say inside AWS. The application probably has other users in addition to Winnie. It's an important application to the business. If it's down, the staff can't work. If they can't work, they don't generate value to the business and of course, this costs the business money. If we have a failure, it means that the system is now suffering an outage, it's not available. System availability is generally expressed in the form of a percentage of uptime. So we might have 99.9 or three nines and this means that we can only have 8.77 hours of downtime per year. Imagine only being able to take a system down for 8.77 hours a year, that's less than one hour per month. It gets worse though, some systems need even higher levels of availability. We've got 99.999% availability or five nines and this only allows for 5.26 minutes per year of downtime. That means for all outages during a year, you have 5.26 minutes. That includes identifying that there's an outage, identifying the cause, devising a solution, and implementing a fix. An outage in this context is defined as something which impacts that server, so impacts your users.

      Now, fixing Winnie's application quickly can be done by swapping out the compute resource, probably a virtual server. Rather than using time to diagnose the issue, if you have a process ready to replace it, it can be fixed quickly and probably in an automated way, or you might improve this further by having two servers online constantly, one active and one standby. In the event of a failure, customers would move to the standby server with very close to zero downtime. But, and this is a key factor about high availability, when they migrate from the active server to the standby server, they might have to re-login or might have some small disruption. For high availability, user disruption, while not being ideal, is okay. It can happen because high availability is just about minimizing any outages.

      Now, this might explain it a little better. This is a real-world example of something which has high availability built in. It's a four by four. If you were driving in the desert with a normal urban-grade car and it got a flat tire, would you have a spare? Would you have the tools ready to repair it as quickly as possible? In a desert, an outage or delay could have major impacts. It's risky and it could impact getting to your destination. So an example of high availability is to carry a spare wheel and the tools required to replace it. You would of course, need to spend time changing the tire, which is a disruption, but it could be done and it minimizes the time that you're out of action. If you don't have a spare tire, then you'd need to call for assistance, which would substantially increase the time you're out of action. So, high availability is about keeping a system operational. It's about fast or automatic recovery of issues. It's not about preventing user disruption. While that's a bonus, a highly available system can still have disruption to your user base when there is a failure.

      Now, high availability has costs required to implement it. It needs some design decisions to be made in advance and it requires a certain level of automation. Sometimes, high availability needs redundant servers or redundant infrastructure to be in place ready to switch customers over to in the event of a disaster to minimize downtime.

      Now, let's take this a step further and talk about fault tolerance and how it differs from high availability. When most people think of high availability, they're actually mixing it up with fault tolerance. Fault tolerance in some ways is very similar to high availability, but it is much more. Fault tolerance is defined as the property that enables a system to continue operating properly in the event of a failure of some of its components, so one or more faults within the system. Fault tolerance means that if a system has faults, and this could be one fault or multiple faults, then it should continue to operate properly, even while those faults are present and being fixed. It means it has to continue operating through a failure without impacting customers.

      Imagine a scenario where we have somebody injured, so we've got Dr. Abbie and she's been told that she has an urgent case of an injured patient and we'll call this patient, Mike. Mike has been rushed to the hospital after injuring himself running. He's currently being prepped for a surgical procedure and is in the operating room and currently under general anesthetic. While he's unconscious, he's being monitored and this monitoring system indicates when to reduce or increase the levels of anesthetic that Mike gets. It's critical that this server is not to be interrupted ever. The system uses underlying infrastructure on-premises at the hospital. Now, in the event of a system failure, if it was just a highly available system, the server could be replaced or another server could be included in an active standby architecture. In either case, the swap between the servers would cause a system error, a disruption. However quick the fix, however small that disruption, in certain situations like this, any disruption can be life-threatening. This is an example of a situation where high availability isn't enough. Fault tolerance systems are designed to work through failure with no disruption. In this example, we might have the system's monitor communicating with two servers at the same time in an active, active configuration. The monitor is connected to both servers all of the time. So this is not just a simple fail-over configuration. If a server failed, it would drop down to just communicating with the remaining server and as long as one server remains active, the system is fully functional. Now, we could take this further adding a second monitoring system, itself with connections to both servers. That way, one monitor can fail, one server can fail and still the service would continue uninterrupted. We could even eliminate the human dependency in the system and add an extra surgeon, Dr. Abbie's twin.

      Most people think that HA means operating through failure, it's not. HA is just about maximizing uptime. Fault tolerance is what means to operate through failure. Fault tolerance can be expensive because it's much more complex to implement versus high availability. High availability can be accomplished by having spare equipment, so standby, physical or virtual components. As long as you automate things and have these spare components ready to go, you can minimize outages. With fault tolerance, it's about more than that. You first need to minimize outages, which is the same as HA, but then you also need to design the system to be able to tolerate the failure, which means levels of redundancy and system components, which can route traffic and sessions around any failed components.

      Now remember the example I used for high availability, the four by four in the desert. There are situations where we can't pull over to the side of the road and change a component. An example of this is a plane, which is in the air. A plane needs to operate through systems failure, so through an engine failure, for example. If an engine fails, the plane can't simply stop and effect repairs. So, a plane comes with more engines than it needs. It comes with duplicate electronic systems and duplicate hydraulic systems, so that when it has a problem, it just carries on running until it can safely land and effect repairs. AWS is no exception to this. Systems can be designed to only maximize uptime, which is high availability, or they can be designed for mission or life critical situations and so, designed to operate through that failure, which is fault tolerance.

      As a solutions architect, you need to understand what your customer requires. A customer might say that they need HA or fault tolerance while not understanding the difference. Fault tolerance is harder to design, harder to implement and costs much more. Implementing fault tolerance when you really needed high availability simply means you're wasting money. It costs more, and it takes longer to implement. But the reverse, implementing high availability when you need fault tolerance, means that you're potentially putting life at risk. A highly available plane is less than ideal. Understand the difference, if you don't, it can be disastrous.

      So, let's move on to the final concept, which is disaster recovery. The definition of disaster recovery is a set of policies, tools, and procedures to enable the recovery or continuation of vital technology infrastructure and systems following a natural or human-induced disaster. So, while high availability and fault tolerance are about designing systems to cope or operate through disaster, disaster recovery is about what to plan for and do when disaster occurs, which knocks out a system. So, if high availability and fault tolerance don't work, what then? What if your building catches fire, is flooded or explodes? Disaster recovery is a multiple-stage set of processes. So given a disaster, it's about what happens before, so the pre-planning and what happens afterwards, the DR process itself.

      The worst time for any business is recovering in the event of a major disaster. In that type of environment, bad decisions are made, decisions based on shock, lack of sleep, and fear of how to recover. So, a good set of DR processes need to preplan for everything in advance. Build a set of processes and documentation, plan for staffing and physical issues when a disaster happens. If you have a business premises with some staff, then part of a good DR plan might be to have a standby premises ready and this standby premises can be used in the event of a disaster. That way, done in advance, your staff unaffected by the disaster, know exactly where to go. You might need space for IT systems or you might use a cloud platform, such as AWS as a backup location, but in any case, you need the idea of a backup premises or a backup location that's ready to go in the event of a disaster.

      If you have local infrastructure, then make sure you have resilience. Make sure you have plans in place and ready during a disaster. This might be extra hardware sitting at the backup site ready to go, or it might be virtual machines or instances operating in a cloud environment ready when you need them. A good DR plan means taking regular backups, so this is essential. But the worst thing you can do is to store these backups at the same site as your systems, it's dangerous. If your main site is damaged, your primary data and your backups are damaged at the same time and that's a huge problem. You need to have plans in place for offsite backup storage. So, in the event of a disaster, the backups can be restored at the standby location. So, have the backups of your primary data offsite and ready to go and make sure that all of the staff know the location and the access requirements for these backups.

      Effective DR planning isn't just about the tech though, it's about knowledge. Make sure that you have copies of all your processes available. All your logins to key systems need to be available for the staff to use when they're at this standby site. Do this in advance and it won't be a chaotic process when an issue inevitably occurs. Ideally, you want to run periodic DR testing to make sure that you have everything you need and then if you identify anything missing, you can refine your processes and run the test again. If high availability is a four-by-four, if fault tolerance are the resilient systems on large planes, then effective DR processes are pilot or passenger ejection systems. DR is designed to keep the crucial and non-replaceable parts of your system safe, so that when a disaster occurs, you don't lose anything irreplaceable and can rebuild after the disaster. Historically, disaster recovery was very manual. Because of cloud and automation, DR can now be largely automated, reducing the time for recovery and the potential for any errors.

      As you go through the course, I'm going to help you understand how to implement high availability and fault tolerance systems in AWS using AWS products and services. So, you need to understand both of these terms really well and disaster recovery. So in summary, high availability is about minimizing any outages, so maximizing system availability. Fault tolerance extends this, building systems which operate through faults and failures. Don't confuse the two. Fault tolerance is much more complex and expensive. It takes a lot more time and effort to implement and manage. I'll help you as we go through the course by identifying how to implement systems which are highly available and how to implement systems which are fault tolerant. AWS provides products and services which help with both of those or just help with one or the other and you need to know the difference. Disaster recovery is how we recover. It's what we do when high availability and fault tolerance don't work and AWS also has many systems and features which help with disaster recovery and one of the things that the exam tests will be your knowledge of how quickly you can recover and how best to recover, given the various different products and services and I'll highlight all of this as we go through the course. At this point, that's everything I wanted to cover, so thanks for listening. Go ahead, complete this video and when you're ready, I'll see you in the next.

    1. Welcome back. In this lesson, I'm going to be covering something that will make complete sense by the end of the course. I'm introducing it now because I want you to be thinking about it whenever we're talking about AWS products and services. The topic is the shared responsibility model. The easiest way to explain this is visually, so let's jump in.

      Remember earlier in the course when I talked about the various different cloud service models? In each of these models, there were parts of the infrastructure stack that you were responsible for as the customer, and parts of the infrastructure stack that the vendor or provider were responsible for. With IaaS, for example, the company providing the IaaS product, so AWS in the case of EC2, they're responsible for the facilities, the AWS data centers, the infrastructure, so storage and networking, the servers, so EC2 hosts, and the hypervisor that allows physical hardware to be carved up into independent virtual machines. You as the customer manage the operating system, any containers, any run times, the data on the instance, the application, and any ways in which it interfaces with its customers. This is an example of a set of shared responsibilities. Part of the responsibilities lie with the vendor, and part lie with you as the customer.

      The AWS shared responsibility model is like that, only applying to the wider cloud platform from a security perspective. It's AWS' way of making sure that it's clear and that you understand fully which elements you manage and which elements it manages. At a high level, AWS are responsible for the security of the cloud. You as a customer are responsible for the security in the cloud. Now let's explore this in a little bit more detail because it will help you throughout the course and definitely for the exam.

      Now I've covered the AWS infrastructure at a high level in a previous lesson. AWS provides these to you as a service that you consume. So AWS are responsible for managing the security of the AWS regions, the Availability Zones, and the edge locations. So the hardware and security of the global infrastructure. You have no control over any of that and you don't need to worry about it. It's the "of the cloud" part, and so it's AWS' responsibility. The same holds true for the compute storage databases and networking which AWS also provide to you. AWS manage the security of those components. In addition, any software which assists in those services, AWS manage all of this part of the stack. So the hardware, the regions, the global network, the compute storage database, and networking services, and then any software that is used to provide that service, AWS manage that end-to-end.

      If you consume a service from AWS, they handle the provisioning and the security of that thing. So take EC2 as an example. The region and the Availability Zone that the instance run in, that's AWS' responsibility. The compute, the storage, the underlying databases and networking for that service, from a security perspective, that's AWS' responsibility. The software, so the user interface, the hypervisor, that's handled by AWS. Now you accept responsibility for the operating system upwards. What does that include? It means things like the client-side data encryption, integrity and authentication; server-side encryption; network traffic protection. If your application encrypts its data, you manage that. If your server uses SSL certificates, you manage those. If you encrypt server-to-server communications, then you also handle that. You're also responsible for the operating system, networking, and any local firewall configuration. You're responsible for applications, identity and access management to things that you will need to implement, manage and control. And then any customer data. So any data that runs in this stack, you need to manage it, secure it, and ensure that it's backed up.

      This might seem like a pretty abstract concept. You might be wondering, does it actually benefit you in the exam? I'd agree with you to a point. When I was doing my AWS studies, I actually didn't spend much time on the shared responsibility model. But what I found is when I sat the exam, I did feel as though it could have benefited me to start learning about it early on when I was first starting my studies. If you keep the shared responsibility in mind as we're going through the various different AWS products, you'll start building up an idea of which elements of that product AWS manage, and which elements you're responsible for. When it comes to deploying an EC2 instance into a VPC or using the Relational Database Service to deploy and manage a database inside a VPC, you need to know which elements of that you manage and which elements AWS manage.

      I'll be referring back to this shared responsibility model fairly often as we go through the course, so you build up this overview of which elements you need to worry about and which are managed by AWS. If possible, I would suggest that you either print out the shared responsibility model and put it on your desk as you're studying, or just make sure you've got a copy that you can refer back to. It becomes important to understand it at this high level. I'm not going to use any more of your time on this topic. I just wanted to introduce it. I promise you that I'm not going to be wasting your time by talking about things which don't matter. This will come in handy. This is definitely something that will help you answer some questions.

      That's all I wanted to cover for now. It's just a foundation, and I don't want to bore you with too much isolated theory. Try to keep this in mind as you go through the rest of the course. For now, this is the level of detail that you need. That's everything I wanted to cover. Go ahead, complete this lesson. When you're ready, move on to the next.

    1. Welcome back. In this lesson, I want to talk about CloudWatch, a core product inside AWS used for operational management and monitoring. CloudWatch performs three main jobs: it collects and manages operational data, monitors metrics, and performs actions based on these metrics.

      CloudWatch collects and manages operational data generated by an environment, including performance details, nominal operations, and logging data. It can be considered three products in one: CloudWatch, CloudWatch Logs, and CloudWatch Events.

      Firstly, CloudWatch allows the collection, monitoring, and actions based on metrics related to AWS products, applications, or on-premises systems. Metrics include data such as CPU utilization, disk space usage, or website traffic. CloudWatch can gather metrics from AWS, on-premises environments, or other cloud platforms using a public internet connection. Some metrics are gathered natively by CloudWatch, while others require the CloudWatch Agent, especially for monitoring non-AWS environments or specific processes on AWS instances.

      CloudWatch provides a user interface, command line interface, or API to access and manage this data. The second part of CloudWatch, CloudWatch Logs, handles the collection, monitoring, and actions based on logging data from various sources like Windows event logs, web server logs, and more. For custom logs or non-AWS systems, the CloudWatch Agent is also needed.

      The third part is CloudWatch Events, which functions as an event hub. It generates events based on AWS service actions (e.g., starting or stopping an EC2 instance) and can also create scheduled events for specific times or days.

      The core concepts of CloudWatch include namespaces, metrics, datapoints, and dimensions. A namespace is a container for monitoring data, helping to organize and separate different areas of data. AWS uses a reserved namespace format (e.g., AWS/EC2 for EC2 metrics), while you can create custom namespaces for your data. Metrics are collections of related data points in a time-ordered structure, such as CPU utilization. Each datapoint includes a timestamp and value. Dimensions, which are name-value pairs, help separate and identify data within a metric, like distinguishing datapoints from different EC2 instances.

      CloudWatch also uses alarms to take actions based on metrics. Alarms can be in an OK state (indicating no issues), an ALARM state (indicating a problem), or an INSUFFICIENT_DATA state (indicating not enough data to assess). Actions could include notifications or more complex responses. You’ve already seen an example of this with the billing alarm created at the start of the course.

      In the next demo lesson, we’ll provision an EC2 instance, let it run, and then create an alarm to monitor CPU usage, providing practical exposure to how CloudWatch works.

      Thanks for watching. Complete this video and join me in the demo when you’re ready.

    1. Welcome back, and in this lesson, I want to talk about AWS CloudFormation. I'm going to be brief because learning CloudFormation is something which will happen throughout the course, as we'll be using it to automate certain things. Before we dive in, I'll introduce the concepts you'll need and give you a chance to experience a simple practical example.

      CloudFormation is a tool which lets you create, update, and delete infrastructure in AWS in a consistent and repeatable way using templates. Rather than creating and updating resources manually, you create a template, and CloudFormation will do the rest on your behalf.

      At its base, CloudFormation uses templates. You can use a template to create AWS infrastructure using CloudFormation. You can also update a template and reapply it, which causes CloudFormation to update the infrastructure, and eventually, you can use CloudFormation to delete that same infrastructure. A CloudFormation template is written either in YAML or JSON. Depending on your experience, you might be familiar with one or both of these. If you haven't touched YAML or JSON before, don't worry. They achieve the same thing, and it's easy to convert between them. You might get to pick which one to use when writing templates, or your business might have a preference. It's mostly a matter of personal preference. Most people in the AWS space like one and dislike the other, though very few people like both. I am one of those who likes both. I started my AWS career using JSON but have come to appreciate the extra functionality that YAML offers. However, YAML can be easier to make mistakes with because it uses white spaces to indicate which parts belong to which other parts. Since spaces are not always visible, it can be a problem for less experienced engineers or architects. If I have to pick one, I'll use YAML. So for the rest of this lesson, I'll focus on YAML.

      I want to quickly step through what makes a template, the components of a template, and then discuss the architecture of CloudFormation before moving on to a demo. All templates have a list of resources, at least one. The resources section of a CloudFormation template tells CloudFormation what to do. If resources are added, CloudFormation creates them. If resources are updated, CloudFormation updates them. If resources are removed from a template and that template is reapplied, then physical resources are removed. The resources section of a template is the only mandatory part of a CloudFormation template, which makes sense because without resources, the template wouldn't do anything. The simple template that we'll use in the demo lesson immediately following this one has resources defined in it, and we'll step through those and evaluate exactly what they do.

      Next is the description section. This is a free text field that lets the author of the template add a description, as the name suggests. Generally, you would use this to provide details about what the template does, what resources get changed, and the cost of the template. Anything that you want users to know can be included in the description. The only restriction to be aware of is if you have both a description and an AWSTemplateFormatVersion, then the description needs to immediately follow the template format version. The template format version isn't mandatory, but if you use both, the description must directly follow the template format version. This has been used as a trick question in many AWS exams, so it pays to be aware of this restriction. The template format version allows AWS to extend standards over time. If it's omitted, the value is assumed.

      The metadata in the template is the next part I want to discuss. It has many functions, including some advanced ones. For example, metadata can control how different elements in the CloudFormation template are presented through the console UI. You can specify groupings, control the order, and add descriptions and labels, which helps in managing how the UI presents the template. Generally, the bigger your template and the wider the audience, the more likely it is to have a metadata section. Metadata serves other purposes, which I'll cover later in the course.

      The parameters section of a template allows you to add fields that prompt the user for more information. When applying the template from the console UI, you'll see boxes to type in or select from dropdowns. This can be used to specify things like the size of the instance to create, the name of something, or the number of availability zones to use. Parameters can have settings for valid entries and default values. You'll gain more experience with this as we progress through the course and use CloudFormation templates.

      The next section is mappings, which is another optional section of the CloudFormation template and something we won't use as much, especially when starting with CloudFormation. It allows you to create lookup tables. For example, you can create a mappings table called RegionAndInstanceTypeToAMI, which selects a specific Amazon Machine Image based on the region and environment type (e.g., test or prod). This is something you'll get experience with as the course continues, but I wanted to introduce it at this point.

      Next, let's talk about conditions. Conditions allow decision-making in the template, enabling certain things to occur only if a condition is met. Using conditions involves a two-step process. Step one is to create the condition. For instance, if a parameter is equal to "prod" (i.e., if the template is being used to create prod resources), then you create a condition called CreateProdResources. If the parameter "environment type" is set to "prod," the condition CreateProdResources will be true. Step two is using this condition within resources in the CloudFormation template. For example, a resource called Prodcatgifserver will only be created if the condition CreateProdResources is true. This will only be true if the "environment type" parameter is set to "prod" rather than "test." If it's set to "test," that resource won't be created.

      Finally, outputs are a way for the template to present outputs based on what's being created, updated, or deleted once the template is finished. For example, outputs might return the instance ID of an EC2 instance that's been created, or if the template creates a WordPress blog, it could return the admin or setup address for that blog.

      So, how exactly does CloudFormation use templates? CloudFormation starts with a template. A template contains resources and other elements you'll become familiar with as we use CloudFormation more. Let's take a simple example—a template that creates an EC2 instance. Resources inside a CloudFormation template are called logical resources. In this case, the logical resource is called "instance," with a type of AWS::EC2::Instance. The type tells CloudFormation what to create. Logical resources generally also have properties that CloudFormation uses to configure the resources in a specific way.

      When you provide a template to CloudFormation, it creates a stack, which contains all the logical resources defined in the template. A stack is a living and active representation of a template. One template could create one stack, or several stacks, or anywhere in between. A stack is created when you tell CloudFormation to do something with that template.

      For any logical resources in the stack, CloudFormation makes a corresponding physical resource in your AWS account. For example, if the stack contains a logical resource called "instance," which defines an EC2 instance, the physical resource is the actual EC2 instance created by CloudFormation. It's CloudFormation's job to keep the logical and physical resources in sync. When you use a template to create a stack, CloudFormation scans the template, creates a stack with logical resources, and then creates matching physical resources.

      You can also update a template and use it to update the stack. When you do this, the stack's logical resources will change—new ones may be added, existing ones updated or deleted. CloudFormation performs the same actions on the physical resources, adding, updating, or removing them as necessary. If you delete a stack, its logical resources are deleted, leading CloudFormation to delete the matching physical resources.

      CloudFormation is a powerful tool that allows you to automate infrastructure. For instance, if you host WordPress blogs, you can use one template to create multiple deployments rather than setting up each site individually. CloudFormation can also be part of change management, allowing you to store templates in source code repositories, make changes, get approval, and apply them as needed. It can also be used for one-off deployments.

      Throughout this course, I'll be using CloudFormation to help you implement various things in demo lessons. If a demo lesson requires certain products to function, I might provide a CloudFormation template to set up the base infrastructure. Alternatively, you can use the template to implement the entire demo end-to-end. CloudFormation is super powerful, and you'll get plenty of exposure to it throughout the course.

      Now, that's all the theory I wanted to cover. The next lesson will be a demo where you'll use CloudFormation to create an EC2 instance. Remember in the EC2 demo lesson, where you created an EC2 instance? In the next demo lesson, you'll create a similar EC2 instance using CloudFormation, demonstrating how much quicker and easier it is to automate infrastructure tasks with CloudFormation. So go ahead, complete this video, and when you're ready, join me in the next lesson where we'll demo CloudFormation.

    1. Welcome back. In this lesson, I want to introduce another core AWS service, the simple storage service known as S3. If you use AWS in production, you need to understand S3. This lesson will give you the very basics because I'll be deep diving into a specific S3 section later in the course, and the product will feature constantly as we go. Pretty much every other AWS service has some kind of interaction with S3. So let's jump in and get started.

      S3 is a global storage platform. It's global because it runs from all of the AWS regions and can be accessed from anywhere with an internet connection. It's a public service. It's regional based because your data is stored in a specific AWS region at rest. So when it's not being used, it's stored in a specific region. And it never leaves that region unless you explicitly configure it to. S3 is regionally resilient, meaning the data is replicated across availability zones in that region. S3 can tolerate the failure of an AZ, and it also has some ability to replicate data between regions, but more on that in the S3 section of the course.

      Now S3 might initially appear confusing. If you utilize it from the UI, you appear not to have to select a region. Instead, you select the region when you create things inside S3, which I'll talk about soon. S3 is a public service, so it can be accessed from anywhere as long as you have an internet connection. The service itself runs from the AWS public zone. It can cope with unlimited data amounts and it's designed for multi-user usage of that data. So millions of users could be accessing cute cat pictures added by the Animals for Life Rescue Officers. S3 is perfect for hosting large amounts of data. So think movies or audio distribution, large scale photo storage like stock images, large textual data or big data sets. It could be just as easily used for millions or billions of IOT devices or to store images for a blog. It scales from nothing to near unlimited levels.

      Now S3 is economical, it's a great value service for storing and allowing access to data. And it can be accessed using a variety of methods. There's the GUI, you can use the command line, the AWS APIs or even standard methods such as HTTP. I want you to think of S3 as the default storage service in AWS. It should be your default starting point unless your requirement isn't delivered by S3. And I'll talk more about the limitations and use cases later in this lesson.

      S3 has two main things that it delivers: Objects and Buckets. Objects are the data the S3 stores, your cat picture, the latest episode of Game of Thrones, which you have stored legally, of course, or it could be large scale datasets showing the migration of the koala population in Australia after a major bushfire. Buckets are containers for objects. It's buckets and objects that I want to cover in this lesson as an introduction to the service.

      So first, let's talk about objects. An object in S3 is made up of two main components and some associated metadata. First, there is the object key. And for now you can think of the object key, similar to a file name. The key identifies the object in a bucket. So if you know the object key and the bucket, then you can uniquely access the object, assuming that you have permissions. Remember by default, even for public services, there is no access in AWS initially, except for the account root user.

      Now, the other main component of an object is its value. And the value is the data or the contents of the object. In this case, a sequence of binary data, which represents a koala in his house. In this course, I want to avoid suggesting that you remember pointless values. Sometimes though, there are things that you do need to commit to memory. And this is one of those times. The value of an object, in essence, how large the object is, can range from zero bytes up to five terabytes in size. So you can have an empty object or you can have one that is a huge five TB. This is one of the reasons why S3 is so scalable and so useful in a wide range of situations because of this range of sizes for objects.

      Now, the other components of an object, aren't that important to know at this stage, but just so you hear the terms that I'll use later, objects also have a version ID, metadata, some access control, as well as sub resources. Now don't worry about what they do for now, I'll be covering them all later. For this lesson, just try to commit to memory what an object is, what components it has and the size range for an object.

      So now that we've talked about objects, let's move on and look at buckets. Buckets are created in a specific AWS region. And let's use Sydney or ap-southeast-2 as an example. This has two main impacts. Firstly, your data that's inside a bucket has a primary home region. And it never leaves that region, unless you as an architect or one of your system admins configures that data to leave that region. That means that S3 has stable and controlled data sovereignty. By creating a bucket in a region, you can control what laws and rules apply to that data. What it also means is that the blast radius of a failure is that region.

      Now this might be a new term. What I mean by blast radius is that if a major failure occurs, say a natural disaster or a large scale data corruption, the effect of that will be contained within the region. Now a bucket is identified by its name, the bucket name in this case, koala data. A bucket name needs to be globally unique. So that's across all regions and all accounts of AWS. If I pick a bucket name, in this case, koala data, nobody else can use it in any AWS account. Now making a point of stressing this as it often comes up in the exam. Most AWS things are often unique in a region or unique in your account. For example, I can have an IAM user called Fred and you can also have an IAM user called Fred. Buckets though are different, with buckets, the name has to be totally unique, and that's across all regions and all AWS accounts. I've seen it come up in the exam a few times. So this is definitely a point to remember.

      Now buckets can hold an unlimited number of objects. And because objects can range from zero to five TB in size, that essentially means that a bucket can hold from zero to unlimited bytes of data. It's an infinitely scalable storage system. Now one of the most important things that I want to say in this lesson is that as an object storage system, an S3 bucket has no complex structure. It's flat, it has flat structure. All objects stored within the bucket at the same level. So this isn't like a file system where you can truly have files within folders, within folders. Everything is stored in the bucket at the root level.

      But, if you do a listing on an S3 bucket, you will see what you think are folders. Even the UI presents this as folders. But it is important for you to know at this stage that that's not how it actually is. Imagine a bucket where you see three image files, koala one, two and three dot JPEG. The first thing is that inside S3, there's no concept of file type based on the name. These are just three objects where the object key is koala1.JPEG, koala2.JPEG and koala3.JPEG. Now folders in S3 are represented when we have object names that are structured like these. So the objects have a key, a forward slash old forward slash koala one, two and three dot JPEG. When we create object names like this, then S3 presents them in the UI as a folder called old. So because we've got object names that begin with slash old, then S3 presents this as a folder called old. And then inside that folder, we've got koala one, two, and three dot JPEG.

      Now nine out of 10 times, this detail doesn't matter, but I want to make sure that you understand how it actually works. Folders are often referred to as prefixes in S3 because they're part of the object names. They prefix the object names. As you move through the various stages of your AWS learnings, this is gonna make more and more sense. And I'm gonna demonstrate this in the next lesson, which is a demo lesson.

      Now to summarize buckets are just containers, they're stored in a region, and for S3, they're generally where a lot of permissions and options are set. So remember that buckets are generally the default place where you should go to, to configure the way the S3 works.

      Now, I want to cover a few summary items and then step through some patterns and anti-patterns for S3, before we move to the demo. But first an exam powerup. These are things that you should try to remember and they will really help in the exam. First bucket names are globally unique. Remember that one because it will really help in the exam. I've seen a lot of times where AWS have included trick questions, which test your knowledge of this one. If you get any errors, you can't create a bucket a lot of the time it's because somebody else already has that bucket name.

      Now bucket names do have some restrictions. They need to be between 3 and 63 characters, all lower case and no underscores. They have to start with a lowercase letter or a number, and they can't be formatted like IP addresses. So you can't have 1.1.1.1 as your bucket name. Now there are some limits in terms of buckets. Now limits are often things that you don't need to remember for the exam, but this is one of the things that you do. There is a limit of a hundred buckets that you can have in an AWS account. So this is not per region, it's for the entire account. There's a soft limit of 100 and a hard limit so you can increase all the way up to this hard limit using support requests, and this hard limit is a thousand.

      Now this matters for architectural reasons. It's not just an arbitrary number. If you're designing a system which uses S3 and users of that system store data inside S3, you can implement a solution that has one bucket per user if you have anywhere near this number of users. So if you have anywhere from a hundred to a thousand users or more of a system, then you're not gonna be able to have one bucket per user because you'll hit this hard limit. You tend to find this in the exam quite often, it'll ask you how to structure a system, which has potentially thousands of users. What you can do is take a single bucket and divide it up using prefixes, so those folders that aren't really folders, and then in that way, you can have multiple users using one bucket. Remember the 100/1000, it's a 100 soft limit and a 1000 hard limit.

      You aren't limited in terms of objects in a bucket, you can have zero to an infinite number of objects in a bucket. And each object can range in size from zero bytes to 5 TB in size. And then finally, in terms of the object structure, an object consists of a key, which is its name and then the value, which is the data. And there are other elements to an object which I'll discuss later in the course, but for now, just remember the two main components, the key and the value. Now, all of these points are worth noting down, maybe make them into a set of flashcards and you can use them later on during your studies.

      S3 is pretty straightforward and that there tend to be a number of things that it's really good at and a fairly small set of things that it's not suitable for. So let's take a look. S3 is an object storage system. It's not a file storage system, and it's not a block storage system, which are the other main types. What this means is that if you have a requirement where you're accessing the whole of these entities, so the whole of an object, so an image, an audio file, and you're doing all of that at once, then it's a candidate for object storage. If you have a Window server which needs to access a network file system, then it's not S3 that needs to be file-based storage. S3 has no file system, it's flat. So you can't browse to an S3 bucket like you would a file share in Windows. Likewise, it's not block storage, which means you can't mount it as a mount point or a volume on the Linux or Windows. When you're dealing with virtual machines or instances, you mount block storage to them. Block storage is basically virtual hard disks. In EC2, you have EBS, which is block storage. Block storage is generally limited to one thing accessing it at a time, one instance in the case of EBS. S3 doesn't have that single user limitation and it's not block storage, but that means you can't mount it as a drive.

      S3 is great for large scale data storage or distribution. Many examples I'll show you throughout the course will fit into that category. And it's also good for offloading things. If you have a blog with lots of posts and lots of images or audio or movies, instead of storing that data on an expensive compute instance, you can move it to an S3 bucket and configure your blog software to point your users at S3 directly. You can often shrink your instance by offloading data onto S3. And don't worry, I'll be demoing this later in the course. Finally, S3 should be your default thought for any input to AWS services or output from AWS services. Most services which consume data and or output data can have S3 as an option to take data from or put data to when it's finished. So if you're faced with any exam questions and there's a number of options on where to store data, S3 should be your default. There are plenty of AWS services which can output large quantities of data or ingest large quantities of data. And most of the time, it's S3, which is an ideal storage platform for that service.

      Okay time for a quick demo. And in this demo, we're just gonna run through the process of creating a simple S3 bucket, uploading some objects to that bucket, and demonstrating exactly how the folder functionality works inside S3. And I'm also gonna demonstrate a number of elements of how access and permissions work with S3. So go ahead and complete this video, and when you're ready join me in the next, which is gonna be a demo of S3.

    1. Welcome back. In this lesson, we will introduce an essential AWS service, the Elastic Compute Cloud (EC2). EC2 is a critical service you need to understand for the AWS exam and for designing real-world AWS solutions, as it is the default compute service within AWS. EC2 provides access to virtual machines known as instances. When deploying compute resources that require an operating system, runtime environment, database dependencies, applications, and application interfaces, EC2 is the service to use. It should be the starting point for any compute requirement in AWS.

      Let's review some important features of EC2 before discussing architectural points. EC2 is an Infrastructure as a Service (IaaS), providing access to virtual machines called EC2 instances. As an IaaS, the unit of consumption is the instance, which is an operating system configured with allocated resources. EC2 is a private AWS service, meaning it runs in the private AWS zone by default. An EC2 instance is configured to launch into a single VPC subnet, set during instance launch, and requires configuration for public access if needed.

      With the default VPC, public access configuration is handled automatically. For custom VPCs, you need to manage this configuration. An EC2 instance's availability zone (AZ) resilience means that if the AZ fails, the instance will likely fail too. Understanding this is crucial for the course and the exam. When launching an instance, you can choose from various sizes and capabilities, influencing the resources and additional features like GPUs, advanced storage, or networking. Some configurations can be changed post-launch.

      As a consumer of EC2, you manage the operating system and upward in the infrastructure stack, while AWS handles virtualization, physical hardware, networking, storage, and facilities. EC2 offers on-demand billing, either by the second or hour, depending on the instance's software, charging only for consumed resources. Instance charges include CPU and memory consumption, storage usage, and extras for any commercial software.

      EC2 instances can use different storage types, including local host storage and Elastic Block Store (EBS), a network storage service. An EC2 instance's state, such as running, stopped, or terminated, indicates its condition. Running instances consume resources and generate charges, while stopped instances do not incur CPU, memory, or networking costs but still generate storage charges. Terminated instances stop all resource usage and delete allocated storage, making this action irreversible.

      An Amazon Machine Image (AMI) is an image of an EC2 instance used to create or derived from an instance. AMIs contain attached permissions, determining which accounts can use them. They can be public, owner-accessible, or explicitly shared with specific AWS accounts. AMIs include the boot volume and a block device mapping, linking volumes to the device IDs expected by the operating system.

      In the next lesson, we will connect to EC2 instances. Windows instances use the Remote Desktop Protocol (RDP) on port 3389, while Linux instances use the SSH protocol on port 22, authenticated with an SSH key pair. When creating an EC2 instance, you select a key pair, download the private key, and AWS retains the public key. For Windows instances, the private key retrieves the local administrator password for RDP access. For Linux instances, the private and public key match enables SSH connection.

      Next, we will demonstrate creating an EC2 instance in the default VPC, assigning a public IPv4 address, and connecting using the discussed methods. Finish this video and join me in the demo lesson when ready.

    1. Welcome back. In this lesson, I want to introduce Virtual Private Clouds known as VPCs. A VPC is the service you will use to create private networks inside AWS that other private services will run from. VPCs are also the service which is used to connect your AWS private networks to your on-premises networks when creating a hybrid environment, or it's the service which lets you connect to other cloud platforms when you're creating a multi-cloud deployment. You will encounter numerous networking and VPC-related questions in the exam, so understanding this service is crucial. This lesson provides a basic introduction, and detailed information will follow later in the course.

      Let's begin with key points. A VPC is a virtual network within AWS, created within an AWS account and a specific region. VPCs are regional services, meaning they are resilient within a region and operate across multiple availability zones. By default, a VPC is private and isolated, allowing communication only within the same VPC unless configured otherwise. There are two types of VPCs: default VPCs and custom VPCs. Each region can have one default VPC, which AWS creates automatically, and you can have multiple custom VPCs. Custom VPCs require detailed configuration and are entirely private by default.

      Custom VPCs are used in most serious AWS deployments because of their flexibility in configuration, size, structure, and connectivity with other VPCs, cloud platforms, and on-premises networks. In contrast, default VPCs are pre-configured by AWS with limited flexibility. This lesson focuses on VPC architecture and the default VPC, while custom VPCs will be covered in depth later.

      VPCs are created within an AWS account and region, such as us-east-1. A region can have multiple custom VPCs, isolated by default unless configured to allow communication. VPCs are resilient, subdivided into subnets located in different availability zones. A default VPC has a set CIDR range of 172.31.0.0/16, which is the same across all regions. Custom VPCs can have multiple CIDR ranges. A VPC's resilience comes from its subnets, each in one availability zone. If an availability zone fails, subnets in other zones continue to operate.

      Key facts about the default VPC include: only one per region, can be deleted and recreated, and has a fixed CIDR range of 172.31.0.0/16. Default VPCs have a predictable structure and come with pre-configured resources like an internet gateway, a default security group, and a network ACL. By default, anything placed in the default VPC subnets is assigned a public IP version four address, making them accessible from the public zone.

      To see the default VPC in action, you can use the AWS Console. In a freshly created AWS account, the default VPC will be present with its CIDR range of 172.31.0.0/16 and subnets in each availability zone of the region. The number of subnets varies by region. You can delete the default VPC and recreate it from the console without support tickets, maintaining the same static structure.

      This lesson provided an overview of the default VPC and its consistent structure. Soon, the course will shift focus to custom VPCs, which are best practice for production environments. The course scenario involving the Animals for Life Organization will illustrate why the default VPC is unsuitable. For now, mark this video as complete, and join me in the next lesson when you're ready.

    1. Welcome back. In this lesson, I'll be discussing how AWS has designed its global infrastructure. While AWS is a global cloud platform, it's actually a collection of smaller groupings of infrastructure connected by a global high-speed network. As solutions architects, we can leverage this to design resilient and highly available systems.

      I'll introduce the infrastructure concepts of an AWS region, an AWS edge location, and an AWS availability zone. I'll also cover the different ways a service can be resilient: globally resilient, regionally resilient, and zone resilient. By the end of this lesson, you'll understand what all of that means. Let's get started.

      At a global level, AWS has created its infrastructure platform as a collection of individual infrastructures located worldwide. The two types of deployment at this global level are AWS regions and AWS edge locations. A region in AWS doesn't directly map onto a continent or a country. It's an area selected by AWS with a full deployment of AWS infrastructure, including compute services, storage, database products, AI, analytics, and more.

      AWS continuously adds regions. At the time of creating this lesson, these include Northern Virginia, Ohio, California, and Oregon in the US, Frankfurt, Ireland, London, and Paris in Europe, and Sao Paulo in South America, among others. Some countries have one region, while larger nations have multiple, depending on customer requirements and size. Regions are geographically spread, enabling solutions architects to design systems that can withstand global-level disasters.

      When interacting with most AWS services, you are interacting with a specific region. For example, Amazon's Elastic Compute Cloud in Northern Virginia is separate from Elastic Compute Cloud in Sydney. AWS can only deploy regions as fast as business and local planning allow, so you might not always have a region in the same town or city as your customers. Therefore, AWS also provides edge locations. Edge locations are much smaller than regions and generally only have content distribution services and some types of edge computing. They are located in many more places than regions, useful for companies like Netflix that need to store content close to their customers for low latency and high-speed distribution.

      Regions and edge locations are commonly used together. A large company like Netflix might run its infrastructure from multiple regions worldwide, but its content could be stored in many edge locations for faster delivery. For example, in Australia, there is an AWS region in Sydney. If a Netflix customer in Melbourne wants to stream a show, it could be streamed from an edge location in Melbourne, providing faster transfer and lower latency.

      AWS has a site to visualize the global AWS network, showing far fewer regions than edge locations, all connected by high-speed networking links. As we go through the course, I'll teach you how to use this private AWS networking for efficient systems deployment in AWS.

      Regions are presented within the AWS Console. For instance, in the EC2 area of the console, you must select a region. However, global services like IAM or Route 53 don't require region selection. Some services are individual deployments in each region, while others operate globally.

      Regions have three main benefits for solutions architects. Firstly, each region is geographically separate, meaning that a problem in one region wouldn't affect others. This isolation provides fault tolerance and stability. Secondly, selecting a region provides geopolitical or governance separation, meaning your infrastructure is subject to the laws and regulations of the region it’s in. AWS commits that data placed in one region won't leave that region unless configured otherwise. Lastly, regions allow location control, enabling you to tune your architecture for performance by placing infrastructure close to your customers.

      Inside regions are availability zones (AZs), which are isolated infrastructure components within a region. Each region has multiple AZs, which can be two, three, four, five, or even six. In Sydney, there are three: ap-southeast-2a, 2b, and 2c. AZs are isolated compute, storage, networking, power, and facilities within a region. If an AZ fails, services in other AZs within the region remain functional. Solutions architects can design systems to distribute components across multiple AZs for resilience.

      An availability zone could be one data center or part of multiple data centers, and AWS does not provide visibility into what constitutes an AZ. Services can be placed across multiple AZs to make them resilient.

      Finally, let's define the resilience of an AWS service. There are three resilience levels: globally resilient, regionally resilient, and AZ resilient. Globally resilient services operate worldwide with a single database, replicating data across multiple regions. Examples include IAM and Route 53. Regionally resilient services operate in a single region with data replicated across multiple AZs in that region, such as RDS databases. AZ resilient services run from a single AZ, and if that AZ fails, the service fails.

      As solutions architects, understanding the resilience level of each AWS service is crucial. This knowledge will help you answer exam questions and become an effective solutions architect.

      These concepts are simple but fundamental. Ensure you understand what a region, edge location, and AZ are, and how services are globally, regionally, or AZ resilient. If needed, rewatch this video. When you're ready, I'll see you in the next lesson.

    1. Welcome back. In this lesson, I want to cover the architecture of public AWS services and private AWS services. This is foundational to how AWS works, from a networking and security perspective. The differences might seem tiny, but understanding them fully will help you grasp more complex network and security products or architectures throughout your studies.

      AWS services can be categorized into two main types: public services and private services. If you don’t have much AWS experience, you might assume that a public service is accessible to everyone, and a private service isn't. However, when you hear the terms AWS private service and AWS public service, it’s referring to networking only. A public service is accessed using public endpoints, such as S3, which can be accessed from anywhere with an internet connection. A private AWS service runs within a VPC, so only things within that VPC, or connected to that VPC, can access the service. For both, there are permissions as well as networking. Even though S3 is a public service, by default, an identity other than the account root user has no authorization to access that resource. So, permissions and networking are two different considerations when talking about access to a service. For this lesson, it's the networking which matters.

      When thinking about any sort of public cloud environment, most people instinctively think of two parts: the internet and private network. The internet is where internet-based services operate, like online stores, Gmail, and online games. If you're at home playing an online game or watching training videos, you’re connecting to the internet via an internet service provider. So this is the internet zone. Then we have the private network. If you’re watching this video from home, your home network is an example of a private network. Only things directly connected to a network port within your house or people with your WiFi password can operate in your personal, private network zone.

      AWS also has private zones called Virtual Private Clouds (VPCs). These are isolated, so VPCs can't communicate with each other unless you allow it, and nothing from the internet can reach these private networks unless you configure it. Services like EC2 instances can be placed into these private zones and, just like with your home network, it can only access the internet, and the internet can only access it if you allow and configure it.

      Many people think AWS is architected with just two network zones: the internet and private zones. But there's actually a third zone: the AWS public zone, which runs between the public internet and the AWS private zone networks. This is not on the public internet but connected to it. The distinction might seem irrelevant, but it matters as you learn more about advanced AWS networking. The AWS public zone is where AWS public services operate, like S3.

      To summarize, there are three different network zones: the public internet, the AWS private zone (where VPCs run), and the AWS public zone (where AWS public services operate). If you access AWS public services from anywhere with a public internet connection, your communication uses the public internet for transit to and from this AWS public zone. This is why you can access AWS public services from anywhere with an internet connection because the internet is used to carry packets from you to the AWS public zone and back again.

      Later in the course, I will cover how you can configure virtual or physical connections between on-premises networks and AWS VPCs, allowing private networks to connect together if you allow it. You can also create and attach an internet gateway to a VPC, allowing private-zone resources to access the public internet if they have a public IP address. This also allows access to public AWS services like S3 without touching the public internet, communicating through the AWS public zone.

      Private resources, such as EC2 instances, can be given a public IP address, allowing them to be accessed from the public internet. Architecturally, this projects the EC2 instance into the public zone, enabling communication with the public internet. Understanding the three different network zones—the public internet, the AWS public zone, and the AWS private zone—is crucial for doing well in the real world and in specific exams. These three network zones become critical as you learn more advanced networking features of AWS.

      That’s everything for this lesson. Complete the video, and when you’re ready, I’ll look forward to you joining me in the next lesson.

  6. Jul 2024
    1. Welcome back.

      I spent the last few lessons going through DNS, helping you, I hope, understand how the system works at an architectural level. In this lesson, I want to finish off and talk about the types of records which can be stored in DNS, and I'll try to keep it quick, so let's get started.

      The first record type that I want to touch on are nameserver records or NS records. I've mentioned these in the previous lessons in this section on DNS. These are the record types which allow delegation to occur in DNS. So we've got the dot com zone, and that's managed by Verisign. This zone will have multiple nameserver records inside it for amazon.com. These nameserver records are how the dot com delegation happens for amazon.com, and they point at servers managed by the amazon.com team. These servers host the amazon.com zone. Inside this one are DNS records such as www, which is how you can access those records as part of DNS.

      Now, of course, the same is true on the other side. The root zone has delegated management of dot com by having nameservers in the root zone point at the servers that host the dot com zone. So nameserver records are how delegation works end-to-end in DNS. Nameservers are hugely important.

      Next up, we have a pair of record types that you will use a lot more often in DNS, and they're A records or AAAA records, and they actually do the same thing. Given a DNS zone, in this example, google.com, these types of records map host names to IP addresses. The difference is the type of IP address. For a given host, let's say www, an A record maps this onto an IP version four address. An AAAA record type is the same, but this maps the host onto an IP version six address. Generally, as an admin or a solutions architect, you will normally create two records with the same name. One will be an A record, and one will be an AAAA record. The client operating system and DNS software on that client can then pick the correct type of address that it wants, either AAAA, if it's capable of IP version six, or just a normal A record, if it's not capable of version six.

      Now next up is the CNAME record type, which stands for canonical name. For a given zone, the CNAME record type lets you create the equivalent of DNS shortcuts, so host to host records. Let's say that we have an A record called server, which points at an IP version four address. It's fairly common that a given server performs multiple tasks. Maybe in this case, it provides ftp, mail, and web services. Creating three CNAMEs and pointing them all at the A server record means that they will all resolve to the same IP version four address. CNAMEs are used to reduce admin overhead. In this case, if the IP version four address of the server changes, it's just the single record to update, the A record, because the three CNAMEs reference that A record, they'll automatically get updated. Now, CNAMEs cannot point directly at an IP address, only other names, and you can expect to see that feature in the exam as a trick question.

      Next is the MX record type, and this is hugely important for how the internet works, specifically how email on the internet works. Imagine if you're using your laptop via your email server and you want to send an email to hi@google.com. MX records are used as part of this process. Your email server needs to know which server to pass the email onto. So we start with the google.com zone. Inside this zone, we have an A record with the name mail, and this is pointing at an IP address. Now it's important to know from the offset that this could be called rabbits or apple or fluffy; the name isn't important to how email works using MX records. In this case, the A record is just called mail, but it doesn't matter.

      Now also inside the google.com zone is a collection of MX records, in this example, two records. MX records have two main parts, a priority and a value, and I'll revisit the priority soon. For now, let's focus on the values. The value can be just a host, as with the top example. So mail here is just mail. That's just a host. If it's just a host and we can tell that by the fact that it's got no dot on the right, it's assumed to be part of the same zone that it's in. So mail here actually means mail.google.com. It's the mail host inside the google.com zone. If you include a dot on the right, this means it's a fully qualified domain name. And so it can either point to the host inside the same zone or something outside that zone, maybe Office 365 if Google decided Microsoft's mail product was better.

      The way that MX records are used is that our email server looks at the two addresses on the mail, so hi@google.com, and it focuses on the domain, so google.com. It then does an MX query using DNS on google.com. This is the same process as any other record type, so it talks to the root first, then dot com, then google.com, and then it retrieves any MX records. In this case, two different records. Now, this is where the priority value is used to choose which record to use. Lower values for the priority field are actually higher priority. So in this example, mail is used first and then mail.other.domain is only used if mail isn't functional. If the priority is the same, then any of them could be selected. Whichever is used, the server gets the result of the query back and it uses this to connect to the mail server for google.com via SMTP and it uses this protocol to deliver the mail. So in summary, an MX record is how a server can find the mail server for a specific domain. MX records are used constantly. Whenever you send an email to a domain, the server that is sending the email on your behalf is using DNS to do an MX lookup and locate the mail server to use.

      The last record type that I want to talk about is a TXT record, also known as a text record. Text records allow you to add arbitrary text to a domain. It's a way in which the DNS system can provide additional functionality. One common usage for a TXT record type is to prove domain ownership. Let's say for the Animals for Life domain, we want to add it to an email system, maybe Google Mail or Office 365 or Amazon WorkMail. Whatever system we use to host our email might ask us to add a text record to the domain, containing a certain piece of text data. So let's say that the random text that we need to add is "cats are the best." Then our administrator would add a record inside this domain with that text data. And once our admin has done that, the external party, so the Google email system, would query that text data, make sure that it matches the value that they're expecting. And if it does, that would prove that we own that domain and we can manage it. So text records are fairly important in proving domain ownership, and that's one of the most common use cases that you will use the text record type for. There are other uses for the text record type. It can be used to fight spam. So you can add certain information to a domain indicating which entities are authorized to send email on your behalf. If any email servers receive email from any other servers, then that's a good indication that that email is spam and not authorized.

      So those are the record types that I want to cover. But there's one more concept that I need to discuss before we finish up. And that is DNS TTL or Time To Live. A TTL value is something that can be set on DNS records. It's a numeric value in seconds. Let's look at a visual example. We have a client looking to connect to amazon.com. And so it queries DNS using a resolver server that's hosted at its internet provider. That resolver server talks to the DNS root, which points at the dot com registry authoritative servers. And so the resolver queries those servers. Those authoritative servers for dot com provide the nameservers at the amazon.com zone, and so it goes ahead and queries that. That server hosts and is authoritative for the amazon.com zone, which has a record for www. And so it uses this record to get the IP address and connect to the server.

      This process takes time. This walking the tree process, talking to the root, and then all of the levels to get the eventual result that you need, it is a lengthy process. Getting a result from the authoritative source, so the source that is trusted by DNS, this is known as an authoritative answer. So you get an authoritative answer by talking to a nameserver, which is authoritative for that particular domain. So if I query the nameserver for amazon.com and I'm querying the www record in amazon.com, then I get back what's known as an authoritative answer. And that is always preferred because it's always going to be accurate. It's the single source of truth.

      But using TTL values, the administrator of amazon.com can indicate to others how long records can be cached for, what amount of time is appropriate. In this example, because the admin of amazon.com has set a 3,600 TTL value, which is in seconds, it means that the results of the query are stored at the resolver server for 3,600 seconds, which is one hour. If another client queries the same thing, which is pretty likely for amazon.com, then they will get back a non-authoritative answer. But that answer will be retrieved immediately because it's cached on the resolver server. The resolver server, remember, is hosted probably at our internet provider, and so it's much quicker to access that data.

      So non-authoritative answers are often the same as authoritative answers. Normally things in DNS don't change, and when they don't change, non-authoritative and authoritative is the same thing. But TTL is important for when things change. If you migrate your email service and you have a high TTL value on your MX record, and you change to a provider with a different IP address, then email delivery might be delayed because old IP addresses for those MX records will be cached and they will be used. TTLs are a balance. Low values mean more queries against your nameservers. High values mean fewer queries, but also less control if you need to change records. You can change TTL values before projects and upgrades or you can leave them permanently low. Also, keep in mind that the resolver should obey TTL values, but that's not always the case. It could ignore them. That configuration can be changed locally by the admin at the resolver server. DNS is often the cause of project failures because of TTL values. If you're doing any work that involves changing any DNS records, it's always recommended to lower the TTL value well in advance of the work, sometimes days or weeks in advance, and this will make sure that you have fewer caching issues when you finally do change those records.

      Okay, that's it. That's everything I wanted to cover in this lesson. I've covered the different DNS record types, as well as introduced you to the TTL concept, which is essential to understand if you want to avoid any DNS-related problems. Thanks for listening. Go ahead, complete this video and when you're ready, join me in the next.

    1. Welcome back and in this demo lesson I'm going to step through how you can register a domain using Route 53. Now this is an optional step within the course. Worst case you should know how to perform the domain registration process within AWS and optionally you can use this domain within certain demos within the course to get a more real-world like experience.

      To get started, as always, just make sure that you're logged in to the IAM admin user of the general AWS account which is the management account of the organization. Now make sure that you have the Northern Virginia region selected. While Route 53 is a global service, I want you to get into the habit of using the Northern Virginia region. Now we're going to be using the Route 53 product, so click in the search box at the top of the screen, type Route 53 and then click to move to the Route 53 console.

      Now Route 53, at least in the context of this demo lesson, has two major areas. First is hosted zones and this is where you create or manage DNS zones within the product. Now DNS zones, as you'll learn elsewhere in the course, you can think of as databases which store your DNS records. When you create a hosted zone within Route 53, Route 53 will allocate four name servers to host this hosted zone. And that's important, you need to understand that every time you create a new hosted zone, Route 53 will allocate four different name servers to host that zone. Now the second area of Route 53 is registered domains, and it's in the registered domains area of the console where you can register a domain or transfer a domain in to Route 53.

      Now we're going to register a domain, but before we do that, if you do see any notifications about trying out new versions of the console, then go ahead and click to try out that new version. Where possible, I always like to teach using the latest version of the console UI because it's going to be what you'll be using long-term. So in my case, I'm going to go ahead and click on, try out the new console, depending on when you're doing this demo, you may see this or not. In either case, you want to be using this version of the console UI. So if you are going to register a domain for this course, then you need to go ahead and click register domains.

      The first step is to type the domain that you want into this box. Now, a case study that I use throughout the course is animals for life. So I'm going to go ahead and register a domain related to this case study. So if I type animalsforlive.com and press enter, it will search for the domain and tell us whether it's available. In this case, animalsforlive.com is not available. It's already been registered. In my case, I'm going to use an alternative, so I'm going to try and register animalsforlive.io. Now, I/O domains are one of the most expensive, so if you are registering a domain yourself, I would tend to advise you to look for one of the cheaper ones. I'm going to register this one and it is available.

      Once I've verified that it is available and it's the one I want, we're gonna go ahead and click on select. We can verify the price of this domain for one year, in this case it's 71 US dollars, and then go ahead and click on proceed to check out. Now it's here where you can specify a duration for the domain registration. You can use the default of one year, or alternatively you can go ahead and pick a longer registration period. For this domain I'm going to choose one year and then you can choose whether you want to auto renew the domain after that initial period. In my case I'm going to leave this selected. You'll see a subtotal at the price and then you can click next to move on to the next step.

      Now at this point you need to specify the contact type. In most cases you'll be putting a person or a company but there's also association, public body or reseller. You need to go ahead and fill in all of these details and they do need to be valid details, that's really important. If you are worried about privacy, most domains will allow you to turn on privacy protection, so any details that you enter here cannot be seen externally. Now obviously to keep my privacy intact, I'm going to go ahead and fill in all of these details and I'm going to hide the specifics and once I've entered them all, I'm going to go ahead and click on 'Next' and you should do the same. Again I've hidden my details on the bottom of the screen.

      Route 53 does tell you that in addition to the domain registration cost there is a monthly cost for the hosted zone which will be created as part of this registration. So there is a small monthly cost for every hosted zone which you have hosted using Route 53 and every domain that you have will need one hosted zone. So I'm going to scroll down. Everything looks good, you'll need to agree to the terms and conditions and then click on submit. Now at this point the domain is registering and it will take some time to complete. You may receive a registration email which may include something that you need to do, clicking on a link or some other form of identity verification. You might not get that, but if you do get it, it's important that you do follow all of the steps contained within that email. And if you don't receive an email, you should check your spam folder, because if there are any actions to perform and you don't, it could result in the domain being disabled.

      You can see the status of the domain registration by clicking on "requests" directly below "registered domains". The status will initially be listed as "in progress", and we need this to change to "successful". So pause the video, wait for this status to change, and then you're good to continue. Welcome back, in my case this took about 20 minutes to complete, but as you can see my domain is now registered. So if we go to registered domains you'll be able to see the domain name listed together with the expiration date, the auto renew status, and the status of the transfer lock. Now transfer lock is a security feature, it means the domain cannot be transferred away from route 53 without you disabling this lock.

      Now we're able to see additional details on the domain if we click on the domain name. Now obviously I've hidden my contact information. If you click on the DNSsecKeys tab then it's here where you can configure DNSsec on the domain. We won't be doing anything with that at this stage. One of the important points I want to draw your attention to is the name servers. So I've registered animalsforlife.io and it's these name servers that will be entered into the Animals for Life record within the .io top level domain zone. So these servers are the ones that the DNS system will point at. These currently are set to four Route 53 name servers. And because we've registered the domain inside Route 53, this process is automatic. So a hosted zone is created, four name servers are allocated to host this hosted zone And then those four name servers are entered into our domain records in our top level domain zone.

      This process end-to-end is all automatic. So the four name servers for the animalsforlife.io hosted zone. These are entered into the animalsforlife.io record within the .io top level domain zone. It's all automatic. So if we move to the hosted zone area of the console and then go inside AnimalsForLife.io and then expand the hosted zone details at the top These are the four name servers which are hosting this hosted zone And if you're paying attention You'll note these are the same four servers that are contained within the registered domains Area of the console and these are the same four servers which have been entered into the .io top level domain zone. Now if you ever delete and then recreate a hosted zone It's going to be allocated with four brand new name servers. These name servers will be different than the name servers for the zone which you deleted So if you delete and recreate a hosted zone You'll be given four brand new name servers. In order to stop any DNS problems you'll need to take these brand new name servers and update the items within the registered domains area of the console but again because you've registered the domain within route 53 this process has been handled for you end to end you won't need to worry about any of this unless you delete and recreate the host of zone.

      Now that's everything you need to do at this point if you followed this process throughout this demo lesson you now have an operational domain within the global DNS infrastructure that's manageable within Route 53. Now as I mentioned earlier this is an optional step for the course if you do have a domain registered then you will have the opportunity to use it within various demo lessons within the course. If you don't, don't worry, none of this is mandatory you can do the rest of the course without having a domain. At this point though that is everything I wanted you to do in this demo lesson. Go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome back. And now that I've talked about the fundamentals of DNS from an abstract perspective, I want to bring this back to an AWS focus and talk about Route 53, which is AWS's managed DNS product.

      Okay, let's jump in and get started with a high level product basics, and then I'll talk about the architecture. Route 53 provides two main services. First, it's a service in AWS, which allows you to register domains. And second, it can host zone files for you on managed name servers, which it provides. Now Route 53 is a global service with a single database. It's one of very few AWS services which operates as a single global service. And as such, you don't need to pick a region when using it from the console UI. The data that Route 53 stores or manages is distributed globally as a single set and it's replicated between regions. And so it's a globally resilient service. Route 53 can tolerate the failure of one or more regions and continue to operate without any problems. Now it's one of the most important AWS products. It needs to be able to scale, stay highly performant, whilst remaining reliable, and continue working through failure.

      So let's look at exactly how Route 53 is architected and exactly what it does to provide these two main services. So the first service that I mentioned at the start of this lesson is that Route 53 allows you to register domains. And to do that, it has relationships with all of the major domain registries. Remember from the last lesson that these are the companies which manage the top level domains. They've been delegated this ability by IANA who manage the root zone for DNS. Now these registries, each manage one specific zone. One of them manages the .com zone and/or the .net zone, and another the .io zone, and so on.

      In the next lesson, I'll be demoing how to register a domain that I'll be using for the course scenario. And that domain will be a .org domain. And so one of these relationships is with the .org registry, an organization called PIR. Now, when a domain is registered, a few things happen. First, Route 53 checks with the registry for that top level domain if the domain is available. For this example to keep it simple, let's just assume it is. Then Route 53 creates a zone file for the domain being registered. And remember a zone file is just a database which contains all of the DNS information for a particular domain. In this case, animals4life.org. As well as creating the zone file, Route 53 also allocates name service for this zone. So these are servers which Route 53 creates and manages which are distributed globally and there are generally four of these for one individual zone.

      So it takes this zone file that it's created. And this is known as a hosted zone, using Route 53 terminology, and it puts that zone file onto these four managed name servers. And then as part of registering the domain it communicates with the .org registry. And this is PIR in this case, and liaising with that registry, it adds these name server records into the zone file for the .org top level domain. And the way that it does this is it uses name server records. So these name server records are how PIR delegate the admin of the domain tools. By adding the name server records to the org zone, they indicate that our four name servers are all authoritative for the domain. And that's how a domain is registered using Route 53.

      It's not a complicated process when you simplify it right down. It's simply the process of creating a zone file, creating a number of managed name servers, putting that zone file on those servers, and then liaising with the registry for the top level domain, and getting a name server records added to the top level domain zone, which point back at these servers. Remember, DNS is just a system of delegation.

      So next, let's quickly take a look at zones inside Route 53. So Route 53 provides DNS zones as well as hosting for those zones. It's basically DNS as a service. So it lets you create a manage zone files. And these zone files are called hosted zones in Route 53 terminology, because they're hosted on AWS managed name servers. So when a hosted zone is created, a number of servers are allocated and linked to that hosted zone. So they're essentially one and the same. From Route 53's perspective, every hosted zone also has a number of allocated managed name servers. Now a hosted zone can be public, which means that the data is accessible on the public internet. The name servers for a public hosted zone live logically in the AWS public zone. And this is accessible anywhere with the public internet connection. So they're part of the public DNS system.

      A hosted zone could also be private which means that it's linked to one or more VPCs and only accessible from within those VPCs. And you might use this type of zone if you want to host sensitive DNS records that you don't want to be publicly accessible. A hosted zone hosts DNS records, which I'll be talking about in an upcoming lesson in much more detail because there are many different types of records. Inside Route 53, you'll see records referred to as record sets. Now there is a tiny difference, but for now you can think of them as the same thing.

      Okay, so now it's time for a demo. I know that DNS has been a lot of theory. And so I wanted to show you a domain being registered and the domain that will be registered is the domain that I'll be using for the course scenario which is animals4life.org. So when you're ready to see that, go ahead, complete this video, and join me in the next.

    1. Welcome back and in this demo lesson you're going to get some experience interacting with CloudWatch. So you're going to create an EC2 instance, you're going to cause that instance to consume some CPU capacity and then you're going to monitor exactly how that looks within CloudWatch. Now to do this in your own environment you'll just need to make sure that you're logged into the general AWS account as the IAM admin user and as always make sure that you have the Northern Virginia region selected which is US-East-1. Once you've got those set correctly then click in the search box at the top and type EC2, find the EC2 service and then just go ahead and open that in a brand new tab.

      Now we're going to skip through the instance creation process because you've done that in a previous demo lesson. So just go ahead and click on instances and then Launch Instance. Under Name, I just want you to put CloudWatch Test as the instance name. Then scroll down and then under the Amazon Machine image to use, go ahead and select Amazon Linux. We're going to pick the Amazon Linux 2023 version, so that's the most recent version of this AMI. It should be listed as Free Tier Eligible, so just make sure that's the case. We'll leave the architecture set to 64-bit x86 and scroll down. It should already be set to an instance type which is free tier eligible, in my case t2.micro. We'll be connecting to this instance using ec2 instance connect so we won't be using an SSH key pair. So in this drop down just click and then say proceed without a key pair. We won't need one because we won't be connecting with a local SSH client. Scroll down further still and under Network Settings click on Edit and just make sure that the default VPC is selected. There should only be one in this list but just make sure that it's set as default. Under Subnet we can leave this as No Preference because we don't need to set one. We will need to make sure that Auto Assign Public IP is set to Enable.

      Under create security group for the name and for the description just go ahead and type CloudWatch SG so CloudWatch SG for both the security group name and the description now the default for security group rule should be fine because it allows SSH to connect from any source location and that's what we want scroll down further still and we'll be leaving storage as default remember this is set from the AMI that we pick. Now because this is a CloudWatch lesson, we're going to set something a little bit different. So expand Advanced Details and then scroll down and look for Detailed CloudWatch Monitoring. Now this does come at an additional cost, so you've got a couple of options. You can just watch me do this or you can do this demo without Detailed Monitoring enabled. And if you don't enable this, it will be entirely free, but you might need to wait a little bit longer for things to happen in the demo lesson so keep that in mind.

      What I'm going to do is I'm going to enable detailed CloudWatch monitoring and if we click on info here we can see some details about exactly what that does and we can also open this in a new tab and explore what additional charges apply if we want to enable it. Now in this case I'm going to enable it you don't have to it's not a huge charge but I think for me demoing this to you it's good that I enable it you don't have to you might just have to wait a little bit longer for things to happen in the demo. Now once all of that set just scroll all the way down to the bottom and go ahead and click launch instance. Now this might take a few minutes to create we're first waiting for this success dialog and once that shows we can go ahead and click on view all instances. Go ahead and click refresh until you see the instance it will start off in a pending state with nothing listed under status check. After a few moments this will change status we'll see that it's in a running state and then we need to wait for this to change to two of two status checks before we continue. So go ahead and pause the video wait for your status check to update and once it does we're good to continue.

      Okay so now this has changed to two out of two checks passed and that's good that's what we want so so it should display running on the instant state and then two out of two checks passed under status check. Once this is the case, go ahead and click in the search box at the top and just type CloudWatch, locate the CloudWatch service, and then open that in a brand new tab. This is the CloudWatch console, and it's here where we're going to create a CloudWatch alarm. Now if you see anything about a new UI or new features, you can just go ahead and close down that dialog. Once we're here, go ahead and click on Alarms on the left and then click on all alarms. This will show a list of all the alarms that you've configured within CloudWatch, and currently there aren't any. What we're going to do is to create an alarm. So click on create alarm, and then click on select metric. Once we're on this screen, scroll down, and we're going to be looking for an EC2 metric, because we need to find the CPU utilization metric, which is inside the EC2 namespace. In other words, it comes from the EC2 service. So go ahead and click on EC2, and then we're looking for per instance metrics. So click on per instance metrics, and this will show all of the EC2 instance metrics that we currently have. Now if I scroll through this list, what you'll see is that I have two different instance IDs, because I'm using this account to create all of these demo lessons. In my case, I see previous instances. Now if you're doing this in your account, if you go back to the EC2 Management Console, you can see your instance ID here. Just remember the last four digits of this instance ID, and then go back to the CloudWatch Console. If you have more than one instance listed in CloudWatch, look for the instance ID that ends with the four digits that you just noted down, and then from that list you need to identify CPU utilization. And so I'm going to check the box next to this metric. Now this is the metric that monitors, as the name suggests, CPU utilization on this specific instance ID, which is our CloudWatch test instance. If I scroll up, I'm able to see any data that's already been gathered for this specific instance. And as you can see, it's not a great deal at the moment because we've only just launched this instance. So I'm gonna go ahead and click on Select Metric, and then because we're creating an alarm, it's going to ask us for what metric and conditions we want to evaluate.

      So I'm going to scroll down, and under Conditions, I'm going to pick Static, because I want this alarm to go into an alarm state when something happens to the CPU utilization. So I'm going to ask CloudWatch that whenever the CPU utilization is greater or equal to a specific value than to go into an alarm state. So that value is going to be 15%. So whenever the CPU utilization on this EC2 instance is greater or equal to 15%, then this alarm will go into the alarm state. So I'm gonna go ahead and click on Next. Now you can set this up so that if this alarm goes into an alarm state, it can notify you using SNS. Now that's useful if this is in production usage, but in this case we're not using it in production, so I'm going to go ahead and click on remove. Scroll down to the bottom, there's also other things that you could pick, so you could do an auto scaling action, an EC2 action, or a systems manager action. But we're going to be talking about these in much more detail as we move through the course. For now we're going to keep this simple, it's just going to be a basic alarm which goes into an alarm state or not. So click on next and then under alarm name I'm going to put CloudWatch test and then high CPU and you should do the same. So type that, click on next, scroll down to the bottom and create that alarm.

      Now initially this alarm state will be insufficient data because CloudWatch hasn't yet gathered enough data on the CPU utilization to generate the state. That's fine because we've we've got another thing that we need to do first. So now move back to the EC2 console and we're going to connect into this instance using EC2 Instance Connect. Remember, that's the web-based way to get access to this instance. So over the top of the CloudWatch Test instance, right click and go to Connect. Make sure that EC2 Instance Connect is selected, so click that tab. You can leave everything as default and click on Connect and that will connect you to this EC2 instance. Now at this point, we need to install an application called stress on this EC2 instance. And stress is an application which will put artificial CPU load onto a system. And that's what we want to do in order to see how CloudWatch reacts. To install stress, we're going to run this command. And this next command will use the yum package manager to install the stress utility. So go ahead and run this command and then clear the screen again. Now the stress command can be run by typing stress and what we're going to do is do a double hyphen help just to get the help for this command. So what we're going to do is we're going to run stress and we're going to specify the number of CPUs to use and we want that number to be the same number of virtual CPUs that this instance has. Now a t2.micro has one virtual CPU and so the command that we need to run is stress space hyphen c space 1 and then space and then we're going to use hyphen t which is the timeout command and this specifies how long we want to run this for. So we're going to specify 3600 so hyphen t and then a space 3600 and this will run the stress for 3600 seconds and that's plenty for us to see how this affects the metrics which are being monitored by CloudWatch.

      Now what I want to do before we do that is go back to the CloudWatch console. You might need to refresh if you haven't seen the state update yet. In my case it's already showing as okay. So this means that it's now got access to some data. So click on this alarm and you'll be able to see that currently the CPU started off at very low levels and then it spiked up and potentially in my case that's because we've just installed some software. But note here this red line which indicates the alarm level for this alarm. So if the CPU utilisation, which is in blue, exceeds this red line then this alarm will move from OK to ALARM. And that's what we want to simulate. So go back to the instance and press Enter to run this stress command. And that's going to begin placing high levels of CPU load on this instance and what we'll see over the next few minutes is CloudWatch will detect this additional CPU load and it will cause this alarm to go from OK into an alarm state. So move back to the CloudWatch console and just keep hitting refresh until you see a change in the alarm state. Again this might take a few minutes. What I suggest you do is pause the video and wait for your alarm to change away from OK and then you're good to continue.

      Now in my case this only took a few minutes and as you can see the CPU load reported by this alarm in CloudWatch went from this value here and spiked all the way up to this value which is well above the 15% of the alarm threshold. So the alarm changed from OK to IN alarm based on this excessive CPU and if we keep monitoring this over time you'll see that this trend continues because this CPU is under extremely high load because it's been artificially simulated using the stress utility. Now if we go back to this EC2 instance and press ctrl and C at the same time this will exit out of the stress utility and at this point the artificial CPU load has been removed and the instance will gradually move back down to its normal levels which is very close to zero. So again what you'll see is this may take a few minutes to be reflected inside CloudWatch. So keep refreshing this once you've cancelled the stress utility and wait for the reported CPU utilization to move back down below the alarm value. Again that might take a few minutes so go ahead and pause the video and wait for this blue line to move back under the red line and once it does you should see that the alarm state changes from in alarm to OK again.

      In my case it took a few minutes for the blue line to move below the alarm threshold and then a few more minutes afterwards for the alarm to change from in alarm to OK. But as you can see at this point that's exactly what's happened once the CPU usage goes below the configured threshold value then the alarm changes back to an OK state. And at this point that's everything that I wanted to cover in this demo lesson on CloudWatch. CloudWatch is a topic that I'm going to be going into much more detail later on in the course. This has just been a really brief introduction to the product and how it interacts with EC2. Now at this point the only thing left is to clear up the account and put it back into the same state as it was at the start of this lesson. So to do that go ahead and click on All Alarms, select the CloudWatch Test High CPU Alarm that you created, click on the actions dropdown, select delete, and then confirm that deletion. Then go back to EC2, go to the instances overview, right click on the CloudWatch test instance, making sure that it is the correct instance, so CloudWatch test, and then select terminate instance and confirm that termination. Now that's going to move through a few states, it will start with shutting down, and you need to wait until that instance is in a terminated state. Go ahead and pause the video and wait for your instance to change into terminated.

      Okay so once your instance has terminated on the menu on the left scroll down go to security groups select the CloudWatch SG security group making sure that you do pick the correct one so CloudWatch SG click on actions scroll down delete security groups and click on delete and at that point the account is back in the same state as it was at the start of this demo lesson. So thanks for watching this video. I hope you gained some experience of the CloudWatch product and again we're going to be talking about it in much more detail later in the course. At this point though go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

    1. Welcome back. In this demo lesson, I want to quickly demonstrate how to use CloudFormation to create some simple resources. So before we start, just make sure you're logged in to the general AWS account and that you've got the Northern Virginia region selected. Once you've got that, just move across to the CloudFormation console.

      So this is the CloudFormation console, and as I discussed in the previous lesson, it works around the concepts of stacks and templates. To get started with CloudFormation, we need to create a stack. When you create a stack, you can use a sample template, and there are lots of different sample templates that AWS makes available. You can create a template in the Designer or upload a ready-made template, and that's what I'm going to do. Now, I've provided a template for you to use, linked to this lesson. So go ahead and click on that link to download the sample template file.

      Once you've downloaded it, you'll need to select 'Upload a template file' and then choose 'File'. Locate the template file that you just downloaded; it should be called 'ec2instance.yaml'. Select that and click on 'Open'. Whenever you upload a template to CloudFormation, it's actually uploading the template directly to an S3 bucket that it creates automatically. This is why, when you're using AWS, you may notice lots of buckets with the prefix CF that get created in a region automatically. You can always go ahead and delete these if you want to keep things tidy, but that's where they come from.

      Now, before we upload this, I want to move across to my code editor and step through exactly what this template does. The template uses three of the main components that I've talked about previously. The first one is parameters. There are two parameters for the template: latest AMI and SSH and web location. Let's quickly talk about the latest AMI ID because this is an important one. The type of this parameter is a special type that's actually a really useful feature. What this allows us to do is rather than having to explicitly provide an AMI ID, we can say that we want the latest AMI for a given distribution. In this case, I'm asking for the latest AMI ID for Amazon Linux 2023 in whichever region you apply this template in. By using this style of parameter, the latest AMI ID gets set to the AMI of the latest version of this operating system.

      The final parameter that this template uses is SSH and web location, which is where we can just specify an IP address range that we want to be able to access this EC2 instance. So that's parameters—nothing special, and you'll get more exposure to these as we go through the course. Now we've also got outputs, and outputs are things that are set when the template has been applied successfully. When a stack creates, when it finishes that process, it will have some outputs. I've created outputs so that we get the instance ID, the availability zone that the instance uses—remember EC2 is an AZ service. It’ll also provide the public DNS name for the instance, as well as the public IP address. The way that it sets those is by using what's known as a CloudFormation function.

      So this is ref, and this is going to reference another part of the CloudFormation template. In this case, it's going to reference a logical resource, the EC2 instance resource. Now, get attribute or get att is another function that's a more capable version of ref. With get attribute, you still refer to another thing inside the template, but you can pick from different data that that thing generates. An EC2 instance, by default, the default thing that you can reference is the instance ID, but it also provides additional information: which availability zone it's in, its DNS name, and its public IP. I’ll make sure to include a link in the lesson that details all of the resources that CloudFormation can create, as well as all of the outputs that they generate.

      The main component of course of this template is the resources component. It creates a number of resources. The bottom two, you don’t have to worry about for now. I’ve included them so I can demonstrate the Session Manager capability of AWS. I'll be talking about that much more later in the course, but what I'm doing is creating an instance role and an instance role profile. You won't know what these are yet, but I’ll be talking about them later in the course. For now, just ignore them. The main two components that we're creating are an EC2 instance and a security group for that instance.

      We’re creating a security group that allows two things into this instance: port 22, which is SSH, and port 80, which is HTTP. So it’s allowing two different types of traffic into whatever the security group is attached to. Then we’re creating the EC2 instance itself. We’ve got the EC2 instance, which is a logical resource, the type being AWS::EC2::Instance, and then the properties for that logical resource, such as the configuration for the instance. We’re setting the type and size of the instance, t2.micro, which will keep it inside the free tier. We’re setting the AMI image ID to use, and it's referencing the parameter, and if you recall, that automatically sets the latest AMI ID. We’re setting the security group, which is referencing the logical resource that we create below, so it creates this security group and then uses it on the instance. Finally, we’re setting the instance profile. Now, that’s related to these two things that I’m not talking about at the bottom. It just sets the instance profile, so it gives us the permission to use Session Manager, which I’ll demonstrate shortly after we implement this.

      There’s nothing too complex about that, and I promise you by the end of the course, and as you get more exposure to CloudFormation, this will make a lot more sense. For now, I just want to use it to illustrate the power of CloudFormation. So I’m going to move back to the console. Before I do this, I’m going to go to services and just open EC2 in a new tab. Once you’ve done that, return to CloudFormation and click on next. We’ll need to name the stack. I’m just going to call it CFN demo one for CloudFormation demo one. Here’s how the parameters are presented to us in the UI. The latest AMI ID is set by default to this value because, if we look at the parameters, it’s got this default value for this parameter. Then SSH and web location also has a default value which is set in the template, and that’s why it’s set in the UI. Leave these two values as default. Once you’ve done that, click on next.

      I’ll be talking more about all of these advanced options later on in the course when I talk about CloudFormation. For now, we’re not going to use any of these, so click on next. On this screen, we need to scroll down to the bottom and check this capabilities box. For certain resources that you can create within CloudFormation, CloudFormation views them as high-risk. In this case, we're creating an identity, an IAM role. Don't worry, I'll be talking a lot more about what an IAM role is in the next section of the course. Because it's an identity, because it's changing something that provides access to AWS, CloudFormation wants us to explicitly acknowledge that we’re to create this resource. So it’s prompting us for this capability to create this resource. Check this box, it’s fine, and then click on submit. The stack creation process will begin and the status will show create in progress.

      This process might take a few minutes. You’re able to click on refresh here, so this icon on the top right, and this will refresh the list of events. As CloudFormation is creating each physical resource that matches the logical resources in the template, it’s going to create a new event. For each resource, you’ll see a create in progress event when the creation process starts, and then you’ll see another one create complete when it creates successfully. If there are any errors in the template, you might see red text, which will tell you the nature of that error. But because this is a CloudFormation template that I’ve created, there’ll be no errors. After a number of minutes, the stack itself will move from Create in Progress to Create Complete.

      I refreshed a couple more times and we can see that the Session Manager instance profiles moved into the Create Complete status and straight after that it started to create the EC2 instance. We’ve got this additional event line saying Create in Progress, and the resource creation has been initiated. We’re almost at the end of the process now; the EC2 instance is going to be the last thing that the stack will create. At this point, just go ahead and pause the video and wait until both the EC2 instance and the stack itself move into Create Complete. Once both of those move into Create Complete, then you can resume the video and we’re good to continue.

      Another refresh, and we can see that the EC2 instance has now moved into a Create Complete status. Another refresh and the entire stack, CFN demo 1, is now in the create complete state, which means that the creation process has been completed and for every logical resource in the template, it’s created a physical resource. I can click on the outputs tab and see a list of all the outputs that are generated from the stack. You’ll note how they perfectly match the outputs that are listed inside the template. We’ve got instance ID, AZ, public DNS, and public IP. These are exactly the same as the outputs listed inside the CloudFormation template. You’ll see that these have corresponding values: the instance ID, the public DNS of the instance, and the public IP version 4 address of the instance.

      If I click on the resources tab, we’ll be able to see a list of the logical resources defined in the template, along with their corresponding physical resource IDs. For the EC2 instance logical resource, it’s created an instance with this ID. If you click on this physical ID, it will take you to the actual resource inside AWS, in this case, the EC2 instance. Now, before we look at this instance, I’m going to click back on CloudFormation and just click on the stacks clickable link at the top there. Note how I’ve got one stack, which is CFN demo one. I could actually go ahead and click on create stack and create stack with new resources and apply the same template again, and it would create another EC2 instance. That’s one of the powerful features of CloudFormation. You can use the same template and apply it multiple times to create the same set of consistent infrastructure.

      I could also take this template because it's portable, and because it automatically selects the AMI to use, I could apply it in a different region and it would have the same effect. But I’m not going to do that. I’m going to keep things simple for now and move back to the EC2 tab. Now, the one thing I want to demonstrate before I finish up with this lesson is Session Manager. This is an alternative to having to use the key pair and SSH to connect to the instance. What I’m able to do is right-click and hit Connect, and instead of using a standalone SSH client, I can select to use Session Manager. I’ll select that and hit Connect, and that will open a new tab and connect me to this instance without having to use that key pair.

      Now, it connects me using a different shell than I'm used to, so if I type bash, which is the shell that you normally have when you log into an EC2 instance, that should look familiar. I’m able to run normal Linux commands like df -k to list all of the different volumes on the server, or dmesg to get a list of informational outputs for the server. This particular one does need admin permission, so I’ll need to rerun this with sudo and then dmesg. These are all commands that I could run in just the same way if I was connected to the instance using an SSH client and the key pair. Session Manager is just a better way to do it, but it requires certain permissions to be given to the instance. That’s done with an instance role that I’ll be talking all about later on in the course. That is the reason why my CloudFormation template has these two logical resources, because these give the instance the permission to be able to be connected to using Session Manager. It makes it a lot easier to manage EC2 instances.

      So that’s been a demo of how easy it is to create an EC2 instance using CloudFormation. Throughout the course, we'll be using more and more complex examples of CloudFormation. I’ll be using that to show you how powerful the tool is. For now, it’s a really simple example, but it should show how much quicker it is to create this instance using CloudFormation than it was to do it manually. To finish up this lesson, I’m going to move back to the CloudFormation console. I’m going to select this CloudFormation stack and click on Delete. I need to confirm that I want to do this because it’s telling me that deleting this stack will delete all of the stack resources.

      What happens when I do this is that the stack deletes all of the logical resources that it has, and then it deletes all of the corresponding physical resources. This is another benefit of CloudFormation in that it cleans up after itself. If you create a stack and that creates resources, when you delete that stack, it cleans up by deleting those resources. So if I click on Delete Stack Now, which I will do, it starts a delete process, and that’s going to go ahead and remove the EC2 instance that it created. If I select this stack now, I can watch it do that. I can click on Events, and it will tell me exactly what it’s doing. It’s starting off by deleting the EC2 instance. If I move back to the EC2 console and just hit Refresh, we can see how the instance state has moved from running to shutting down.

      Eventually, once the shutdown is completed, it will terminate that instance. It’ll delete the storage, it will stop using the CPU and memory resources. At that point, the account won’t have any more charges. It wouldn’t have done anyway because this demo has been completely within the free tier allocation because I was using a t2.micro instance. But there we go. We can see the instance state has now moved to terminated. Go back to CloudFormation and just refresh this. We’ll see that it’s completed the deletion of all the other resources and then finished off by deleting the stack itself. So that’s the demonstration of CloudFormation. To reaffirm the benefits, it allows us to do automated, consistent provisioning. We can apply the same template and always get the same results. It’s completely automated, repeatable, and portable. Well-designed templates can be used in any AWS region. It’s just a tool that really does allow us to manage infrastructure effectively inside AWS.

    1. Welcome back and in this demo lesson, I just want you to get some experience working with S3.

      In this demo lesson you're going to create an S3 bucket which is going to be used for a campaign within the Animals for Life organization.

      You're going to get the chance to create the bucket, interact with the bucket, upload some objects to that bucket and then finally interact with those objects.

      Now to get started you'll need to make sure that you're logged in to the IAM

      admin user within the general AWS account. By this point in the course you

      should have a general account and a production account and you need to make

      sure that you're logged in to the general AWS account. As always make sure

      that you're also using the Northern Virginia region which is US-East-1.

      Now assuming that you do have that configuration next you need to move to

      the S3 console and there are a couple of ways that you can do that you can type

      S3 into this find services box if you've previously used the service it will be

      listed under the recently visited services and then finally at the top

      here you can click on the services drop-down and either type S3 into the

      all services box here or locate it in the list of services and click to move

      to the S3 console so I'm going to go ahead and type S3 and then click to move

      to the console. Now when you first arrive at the S3 console you'll be presented

      with a list of buckets within this AWS account. I want to draw specific

      attention to the fact that with S3 you do not have to choose a region with the

      region drop-down. When you create buckets within S3 you have to pick the region

      that that bucket is created in but because S3 uses a global namespace you

      don't have to select a region when using the console. So on this list you will see

      any buckets in any regions within this one single AWS account. You don't have to

      pick the region in advance. So let's go ahead and create an S3 bucket and to do

      that logically enough we click on create bucket. Now to create a bucket you need

      to specify a name and we're creating this bucket for a koala campaign for the

      Animals for Life organization. So we're going to start with Koala Campaign.

      Now because bucket names do need to be unique we can't just leave it at Koala Campaign

      we need to add some random numbers at the end. This is just to make sure that

      the name that you pick is different than the name that I pick and different than

      the name that every other student uses. So just put some numbers after this name.

      I'm going to pick 1-3-3-3-3-3-7. Now there are some rules around bucket

      naming names need to be between 3 and 63 characters they can only consist of

      lowercase letters numbers dots and hyphens they need to begin and end with a

      letter or number they can't be formatted like an IP address and they can't begin

      with X and N and of course they need to be entirely unique now there are some

      specific restrictions or specific rules for naming buckets if you want to use

      certain S3 features. Later in the course I'll be talking about static website

      hosting within S3 and I'll be showing you how you can use a custom domain name

      with an S3 bucket so you can get a domain name to host for example a blog

      or a static website and you can use S3 to host that website and if you want to

      do that then you need to name the bucket name the same as the DNS name that

      you'll be using to access that bucket. But at this point this is just an

      introductory demo so we can leave this as just a standard name. So use Koala

      campaign with some random numbers at the end and that should be good. Now when

      you're creating a bucket you need to specify a region and this is the region

      that this bucket will be placed in. Now I'm going to use US-East-1

      as a default throughout this course and so I do recommend that you pick that to

      create the bucket in. Now if you have any existing buckets within your account and

      you want to copy the settings from those buckets and this will of course just

      save you some time when setting up the bucket then you can click on choose

      bucket and copy the settings from another bucket in your account. Now

      because we're starting this from fresh and we don't have any existing buckets we

      can't use this option. So we need to scroll down and just review what options

      we have. For now we're going to skip past object ownership because this is a

      feature that I'll be discussing in much more detail later in the course. I can't

      really explain this until you have some experience of how the permissions model

      works with S3 so I'll be talking about it in the S3 section of the course. Now

      the first thing that you need to pick when you're creating buckets is this

      bucket settings for block public access. So all S3 buckets by default are

      private. Nobody has permissions to this bucket apart from the account that

      creates the bucket. So in this particular case we're creating it inside the

      general AWS account and so only the general AWS account and the account root

      user of that account have permissions. Now because we've granted the IAM admin

      user full admin permissions then it too has access to this bucket but by default

      nothing else can have access. Now you can make a bucket completely public. You can

      grant access for all users to that bucket including unauthenticated or

      anonymous users. Now that's a security risk because potentially you might have

      sensitive data within that bucket. This is a fail-safe. This means that even if

      you grant completely public access to a bucket then this will block that access.

      And I'm going to be talking about this in much more detail later in the course

      But you need to know that this exists if we untick this option for example

      Even though we are now not blocking all public access

      You still need to grant access to this bucket

      So all this option does is prevent you granting public access if you disable it

      It does not mean that the bucket is public. It just means that you can grant public access to this bucket

      So for this demonstration, we're going to go ahead and untick this option

      Now if you do untick this you'll need to scroll down and check this box just to acknowledge that you understand

      Exactly what you're doing

      So this is a safety feature of s3 that if you're going to remove this fail-safe check then you need to accept responsibility

      It means that if you do mistakenly grant public access to the bucket then potentially information can be exposed.

      Now I'm not going to explain any of these other options because I cover all of them in the S3 section of the course.

      So I'm going to skip past bucket versioning and tags,

      default encryption, and I'm not going to be covering any of these advanced settings.

      Instead, let's just go ahead and click on create bucket.

      At this point you might get an error that a bucket with the same name already exists, and that's fine.

      Remember S3 bucket names need to be globally unique

      And there's obviously a lot of koala campaigns happening in the wild. If you do get the error

      Then just feel free to add extra digits of random to the bucket name

      Then scroll all the way down to the bottom and create the bucket. Once the bucket's created

      You'll see it in the list of buckets

      So there's a column for the name a column for the region

      So you'll be able to see which region this bucket is in

      It will give you an overview of the access that this bucket has so because we unchecked the block public access

      Then it informs us that objects can be public again

      Just to stress this does not mean they are public because s3 buckets are private by default

      This is just telling us that they can be public

      Lastly, we also have the creation date which tells us when the bucket was created.

      So now let's just go ahead and click on the bucket to move inside so we can see additional information.

      Now one thing that I do want to draw your attention to is the Amazon resource name or ARN for this bucket.

      All resources in AWS have a unique identifier, the ARN or Amazon resource name.

      So this is the ARN for the bucket that we've just created

      ARNs have a consistent format. They start with ARN for Amazon resource name

      Then they have the partition for most

      AWS resources in most regions this will always say AWS

      Then you have the service name in this case S3

      Then you have some other values which I'll be talking about later in the course and you can omit those with certain

      services by just putting double colons. These for example might be the region or the account number.

      Now for services where resources are not globally unique

      then obviously you need to specify the region and the account number in order for this name to be globally unique.

      But because S3 buckets by default have to be globally unique

      then we don't have to specify in the ARN either

      the region or the account number. As long as we have the S3 service and the bucket name, we know that this uniquely

      references a resource and that's the key thing about ARNs. ARNs

      uniquely reference one resource within

      AWS. You always know if you have one ARN that it references one particular resource within AWS.

      Now you can use wildcards to reference multiple resources,

      but as a basis it has to reference at least one.

      Now let's just click on the objects tab here

      and this will give us an overview of all of the objects

      which are in this bucket.

      You have a number of tabs here that you can step through.

      We've got a properties tab where you can enable

      bucket versioning, tags, encryption, logging,

      CloudTrail data events, event notifications,

      transfer acceleration, object lock, request to pays,

      and static website hosting.

      and we'll be talking about all of those features in detail

      within the S3 section of the course.

      We'll also be covering permissions in that section

      because you can be very granular

      with the permissions of S3 buckets.

      You can see some metrics about the bucket.

      So this uses CloudWatch,

      which we'll be talking about in detail

      elsewhere in the course.

      You're also able to access management functionality.

      Again, we'll be talking about all of this

      later in the course.

      And then finally you're able to create access points.

      Now access points are some advanced functionality and so we'll be covering this later in the course.

      For now I just want you to get some experience of uploading some objects and interacting with them.

      Now there's a link attached to this lesson which you'll need to go ahead and click and that will download a zip file.

      Go ahead and extract that zip file and it will create a folder.

      And then once you've extracted that into a folder we're good to continue.

      Now the easiest way at this point to upload some objects is to make sure that you've got the objects tab selected

      and then click on upload.

      Now you're able to upload both files and folders to this S3 bucket.

      So let's start off by uploading some files.

      So click on add files.

      Now at this point locate and go inside the folder that you extracted a few moments ago

      and you'll see that there are three image files.

      We've got koala_nom1.jpg, koala_nom2.jpg and koala_zzz.jpg

      Now go ahead and select all three of these JPEG files and click on open.

      You'll see that you have three files in total queued for upload

      and you'll be provided with an estimate of the amount of space that these files will consume.

      Now scrolling down you're told the destination where you'll be uploading these objects to

      So this is the S3 bucket that we've created and this will be different for you.

      This will be your bucket name.

      Now we haven't enabled versioning on this bucket.

      This is a feature which I'll be covering in the S3 section of the course, but

      because we don't have versioning enabled, it means that if we do upload files with

      the same name, then potentially we're going to overwrite other objects in that bucket.

      So we have to accept the risk because we don't have versioning enabled.

      We could overwrite objects if we re-upload ones with the same name.

      In this case that's fine because we're not uploading anything important

      and regardless this bucket is empty so we can't overwrite anything.

      You have the option of enabling versioning where you can just acknowledge the risk.

      Then we can scroll down further still. We need to pick the storage class for the objects.

      This defaults to standard and I haven't covered storage classes in the course yet.

      I'll be doing that within the S3 section, so we're going to accept the default

      and then we're going to skip past all of these options.

      I'll be covering these later in the course and just go ahead and click on Upload.

      And this will upload all of those three objects to the S3 bucket.

      You'll be told whether the upload has been successful or whether it's failed.

      In our case, it's succeeded.

      So we can go ahead and click on Close to close down this dialogue.

      Now when we scroll down, we'll see an overview of the objects within this bucket.

      In our case we only have the three, Koala Nom1, Koala Nom2 and KoalaZZZ.jpg

      We can also create folders within S3. Now of course because S3 is a flat structure

      This isn't actually creating a folder. It's just creating a file which emulates a folder

      So if we create a folder and let's call this folder archive and then click on create folder

      It's not actually creating a folder called archive what it's doing is creating an object with this name

      so archive forward slash

      Now if we click on this archive folder and go inside it we can upload objects into this folder

      So let's go ahead and do that click on upload

      Go to add files and then just pick one of these files. Let's go with koala

      zz.jpg so select that one and click on open and just click on upload. Now what

      we've done is we've uploaded an object into what we see as a folder in this s3

      bucket. If we click on close what this has actually done is it's created an

      object which is called archive/koalazz.jpg. S3 doesn't really have

      folders. Folders are emulated using prefixes and that's important to know as

      you move through the course. Now if we click this at the top to go back to the

      main bucket and we're going to go ahead and open one of these objects. So let's

      pick one of these objects let's use koala_nom1.jpg. This opens an overview

      screen for this particular object and where we see this object URL just go

      ahead and right-click and open that in a new tab. When you open that in a new tab

      you'll be presented with an access denied error. The reason for that is

      you're trying to access this object with no authentication. You're accessing the

      object as an unauthenticated user and as I mentioned earlier all S3 objects and

      all S3 buckets are private by default and that's why you get this access

      denied you won't be able to access this object without authenticating to AWS and

      using that identity to access an object. That's of course unless you grant public

      access to this object which we haven't done and we won't be doing in this

      lesson. So if we close down that tab and instead click on open you might have to

      bypass a pop-up blocker but this time it will open this object and that's because

      we're including authentication in the URL at the top here. So when you click on

      the open button it's opening the object as you it's not opening it as an

      unauthenticated identity so that's important because you have access to

      this bucket you can open the objects using this open button. The same is true

      for the other objects so if we go back to the bucket let's pick koala nom2

      Then click on the open button and again we'll see a koala having some food.

      Go back to the bucket and then let's try koala ZZZ.

      So click on the object, click on open again and now we can see a koala having a well deserved

      rest after his lunch.

      Now that's everything I wanted to cover in this demo lesson.

      It's just been a really high level introduction into how to interact with S3 using the console

      UI.

      be covering S3 in detail later in the course, I just wanted this demo lesson to

      be a very brief introduction. Now what we need to do before we finish this demo

      lesson is to go back to the main S3 console and we need to tidy up by

      deleting this bucket. So deleting buckets within S3 is a two-step process. First we

      need to empty the bucket. So go ahead and select the bucket and click on empty.

      You'll need to either type or copy and paste permanently delete into the box

      and then click on empty and that will remove any objects within the bucket.

      Assuming that's successful go ahead and click on exit and with the bucket still

      selected click on delete and then once you've clicked on delete you'll need to

      copy and paste or type the name of the bucket and finally click on delete

      bucket to confirm that deletion process and that will delete the bucket and your

      account will be back in the same state as it was at the start of this demo

      lesson. Now at this point I hope this has been useful it's just been a really

      basic introduction to S3 and don't worry you'll be getting plenty more theory and

      practical exposure to the product in the S3 section of the course. For now just go

      ahead and complete this video and when you're ready I look forward to you

      joining me in the next.

    1. Welcome back, this is part two of this lesson where we continue immediately from the end of part one. In my case, it took about three minutes for the status check to change to "two out of two checks passed," indicating we're good to continue. AWS runs several checks on an EC2 instance as the final stage of the provisioning process, so ensure an instance has passed these checks before proceeding. You can see the instance has been provisioned into US-East-1C, with its own public IP version 4 address and public IP version 4 DNS name for internet access. Additional details such as the security group name, key name, and launch time are also available. Clicking on the instance reveals more detailed information in the bottom pane across various tabs like Details, Security, Networking, Storage, Status Checks, and Monitoring. To connect to the terminal of this EC2 instance, right-click the instance, select "connect," and choose one of the connection methods. EC2 Instance Connect allows access via a web browser. For this lesson, we focus on connecting using a local SSH client. Ensure you have the local SSH client and the A4L.PEM key file downloaded earlier. In your terminal, navigate to the folder containing the key file and adjust the file's permissions using the chmod 400 command. If on Windows, follow the detailed steps linked in the lesson to correct permissions. After setting permissions, use the provided SSH connection command to connect to the EC2 instance. This command uses the key pair to authenticate your connection. AWS stores the public part of the key pair on the EC2 instance, while you have the private part, allowing secure access. Throughout the course, we will use various methods to connect to EC2 instances, including local SSH clients, Instance Connect, and Session Manager. For Windows EC2 instances, we'll use Remote Desktop, which is beyond the scope of this lesson. To finish up, terminate the instance by selecting it, right-clicking, and choosing "terminate instance." Additionally, delete the security group created for this instance by navigating to the security groups, selecting it, and deleting it once the instance is terminated. Complete these steps and then proceed to the next lesson.

    1. Welcome to this demo lesson where you'll create your first EC2 instance as part of this course. This lesson falls under the AWS Free Tier, so assuming you're using a new account created for this course, there won't be any charges. Ensure you're logged into the general AWS account and have the Northern Virginia region (US-East-1) selected. Once there, navigate to the EC2 console by typing "EC2" in the search box, finding the EC2 service, and clicking to access the console. At the dashboard, the first step is to create an SSH key pair, essential for accessing the EC2 instance. If you don’t already have one, scroll down the left menu, click on Keypairs under Network and Security, and create a key pair. Remember, you'll only have one opportunity to download your part of the key pair, so keep it safe. Name it "A4L" (Animals for Life), choose the key pair type and private key file format based on your operating system (PEM for Mac OS/Linux or modern Windows, PPK for older Windows/using PuTTY), and save it to your default download location.

      Next, return to the dashboard to begin creating your EC2 instance. Click on "Instances" to see all launched instances (you shouldn't have any yet), then click "Launch Instances" to go to the launch instance console. Name the instance "my first EC2 instance" and select the Amazon Machine Image (AMI) for the operating system—use Amazon Linux 2023 for this demo, as it’s free tier eligible. Leave all defaults and choose the instance type, ensuring it’s a free tier eligible type like t2.micro. Select the SSH key pair "A4L" you created earlier.

      Configure network settings by selecting the default VPC and ensure the option to auto-assign a public IP is enabled for internet access. Create a new Security Group named "my_first_instance_sg" to allow SSH from anywhere, understanding you'll need the key pair for access. Configure storage, leaving the default 8GIB GP3 root volume, and expand advanced details if necessary for later use. Launch the EC2 instance and wait for the creation process, which may take a few moments. Refresh to see the instance status change from pending to running, and wait for the status check to complete. This concludes part one of the lesson. Take a break and join part two to continue from here.