Welcome back in this video I want to talk about SSL and TLS.

At a very high level they do the same thing.

SSL stands for Secure Sockets Layer whereas TLS is Transcore Layer Security.

TLS is just a newer and more secure version of SSL.

Now we've got a lot to cover so let's jump in and get started.

TLS and historically SSL provide privacy and data integrity between client and server.

If you browse through this site to Netflix, to your bank and to almost any responsible internet business, TLS will be used for the communications between the client and the server.

TLS performs a few main functions and while these are separate, they're usually performed together and referred to as TLS or SSL.

First, TLS ensures privacy and it does this by ensuring communications made between a client and server are encrypted so that only the client and server have access to the unencrypted information.

When using TLS the process starts with an asymmetric encryption architecture.

If you've watched my encryption 101 video, you'll know that this means that a server can make its public key available to any clients so that clients can encrypt data that only that server can decrypt.

Asymmetric encryption allows for this trustless encryption where you don't need to arrange for the transfer of keys over a different secure medium.

As soon as possible though you should aim to move from asymmetric towards symmetric encryption and use symmetric encryption for any ongoing encryption requirements because computationally it's far easier to perform symmetric encryption.

So part of the negotiation process which TLS performs is moving from asymmetric to symmetric encryption.

Another function that TLS provides is identity verification.

This is generally used so that the server that you think you're connecting to, for example Netflix.com, is in fact Netflix.com.

TLS is actually capable of performing full two-way verification but generally for the vast majority of situations it's the client which is verifying the server and this is done using public key cryptography which I'll talk more about soon.

Finally TLS ensures a reliable connection.

This is a very simple way to do it.

The client can detects against the alteration of data in transit.

If data is altered then the protocol can detect this alteration.

Now in order to understand TLS a little better let's have a look at the architecture visually.

When a client initiates communications with a server and TLS is used there are three main phases to initiate secure communication.

First Cypher suites are agreed, authentication happens and then keys are exchanged.

These three phases start from the point that a TCP connection is active between the client and the server so this is layer four.

And at the end of the three phases there's an encryption communication channel between a client and a server.

This each stage is responsible for one very specific set of functions.

The first stage focuses on Cypher suites.

Now a Cypher suite is a set of protocols used by TLS.

This includes a key exchange algorithm, a bulk encryption algorithm and a message authentication code algorithm or MAC.

Now there are different algorithms and versions of algorithms for each of these and specific versions and types grouped together are known as a Cypher suite.

So to communicate the client and server have to agree a Cypher suite to use.

Now let's step through this visually.

We have a client and a server and at this starting point we already have a TCP connection so TCP segments between the client and the server.

The first step is that the client does a client hello and this contains the SSL or TLS version, a list of Cypher suites that the client supports and other things like a session ID and extensions.

Hopefully at this point the server supports one of the Cypher suites that the client also supports.

If not then the connection fails.

If it does then it picks a specific one and it returns this as part of the server hello.

Now included in this server hello is also the server certificate which includes the server's public key.

Now this public key can be used to encrypt data which the client can send to the server which only the server can decrypt using its private key.

But keep in mind that this is asymmetric encryption and it's really computationally heavy and we want to move away from this as soon as possible.

Now at some point in the past the server has generated a private and public key pair and it's the public part of this which is sent back to the client.

But and this is important part of TLS is ID validation.

If the client just confirmed that the server it's communicating with is valid then you could exploit this.

I could create a server which pretends to be Netflix.com without being Netflix.com and this is suboptimal.

So it's important to understand and I'll talk more about this in a second that part of the functionality provided by TLS is to verify that the server that you're communicating with is the server that it claims to be.

The next step of the TLS process is authentication.

The client needs to be able to validate that the server certificate the server provides is valid, that its public key is valid and as such that the server itself is valid.

To illustrate how this works let's rewind a little from a time perspective.

So the server has a certificate.

Now the certificate you can think of as a document, a piece of data which contains its public key, its DNS name and other pieces of organizational information.

Now there's another entity involved here known as a public certificate authority or CA.

Now there are a few of these run by independent companies and your operating system and browser trust many of these authorities and which ones is controlled by the operating system and browser vendors.

Now at some point in the past our server and let's say this is for Categoram.io created a public and private key pair and in addition it generated a certificate signing request or CSR.

It provided the CSR to one of the public certificate authorities and in return this CA delivered back a signed certificate.

The CA signed a certificate which means that you can verify that the CA signed that certificate.

If your operating system or browser trusts the certificate authority then it means your operating system or browser can verify that the CA that it trusts signed that cert.

This means that your OS or browser trusts the certificate and now the Categoram.io server that we're using as an example has this certificate and that certificate has been provided to the client as part of the server hello in Stage 1 of the TLS negotiation.

In Stage 2 of authentication our client which has the server certificate validates that the public certificate authority signed that certificate.

It makes sure that it was signed by that specific CA, it makes sure that the certificate hasn't expired, it verifies that the certificate hasn't been revoked and it verifies that the DNS name that the browser is using in this case Categoram.io matches the name or the names on the certificate.

This proves that the server ID is valid and it does this using this third party CA.

Next the client attempts to encrypt some random data and send it to the server using the public key within the certificate and this makes sure that the server has the corresponding private key.

This is the final stage of authentication.

If we're at this point and everything is good, the client trusts the server, its ID has been validated and the client knows that the server can decrypt data which is being sent.

It's at this point that we move on to the final phase which is the key exchange phase.

This phase is where we move from asymmetric encryption to symmetric encryption.

This means it's much easier computationally to encrypt and decrypt data at high speeds.

We start this phase with a valid public key on the client and a matching private key on the server.

The client generates what's known as a pre-master key and it encrypts this using the server's public key and sends it through to the server.

The server decrypts this with its private key and so now both sides have the exact same pre-master key.

Now based on the cipher suite that's being used, both sides now follow the same process to convert this pre-master key into what's known as a master secret.

And because the same process is followed on the same pre-master key, both sides then have the same master secret.

The master secret is used over the lifetime of the connection to create many session keys and it's these keys which are used to encrypt and decrypt data.

So at this point both sides confirm the process and from this point onwards the connection between the client and server is encrypted using different session keys over time.

So this is the process that's followed when using TLS.

Essentially we verified the identity of the server that we're communicating with, we've negotiated an encryption method to use, we've exchanged asymmetric for symmetric encryption keys and we've initiated this secure communications channel.

And this process happens each and every time that you communicate with the server using HTTPS.

Now that's everything I wanted to cover within this video so go ahead and complete the video and when you're ready I'll look forward to you joining me in the next.

Thank you.

Welcome back, and in this fundamentals lesson, I want to step you through how to convert decimal numbers into binary and back again, specifically relating to IP version 4 addresses.

Now, if this is something which is new to you, I really suggest taking this slowly and focusing on each step of the process.

If you don't understand something, just pause, think about it, and if needed, restart the video.

I promise, once you can do this process, it will really help you with networking.

The process is easy once you've done it a few times, so I want to explain it as simply as I can at first, and then suggest that you get some practice.

So let's jump in and get started.

Before we start with the actual process, I want to set the scene as to why you need to understand it.

When dealing with IP addresses as a human, you might see something which looks like this.

133.33.33.7.

This is an IP address represented in what's known as dotted decimal notation.

Four decimal numbers ranging from 0 to 255 separated by periods.

So that's what a human sees.

A computer sees this, the same IP address put in binary.

Now, this is actually a 32-bit binary number, specifically four sets of 8 bits.

Now, 8 bits is known as a byte, each byte is also known as an octet, but crucially, it's the same number as the decimal version.

To understand much of how the internet protocol works, so things like IP ranges, prefixes, subnet masks and routing, you have to be able to convert between the two, so decimal and binary.

All of these things only make sense when viewing IP addresses in binary, so when it comes to networking, being able to convert between decimal and binary or vice versa really is a superpower, and that's what I want to teach you in this lesson.

So let's move on.

I want to start with the process of converting a decimal IP address into binary.

This is actually the more complex direction.

Decimal to binary is confusing at first, whereas binary to decimal is much easier.

When you're just learning this process, I find that it's easier to tackle it in byte-sized pieces.

Let's say that you want to convert this IP, so 133.33.33.7, into binary.

Well, each decimal in between the dots, so 133, 33, 33 and 7 is a number between 0 to 255.

So I find it easier, at least initially, to tackle each one of these numbers individually, working left to right.

So that's what we're going to do in this lesson.

I'm going to step through the maths involved to convert 133 and 33 into binary, and then I'm going to ask you to follow the same process for the last two numbers.

Now, before I step through this process, let me introduce this table.

This helps us with binary maths because it tells us what the decimal value is for each position in binary.

Now, this table has eight positions, shown as 1 to 8 on the table, and so it works for 8-bit binary numbers.

Remember, each part of an IP address, each number between the periods, is actually an 8-bit binary number.

Now, each position in this table has a value, so 128, then 64, then 32, 16, 8, 4, 2 and finally 1.

It means a binary 1 in that position has the associated value in decimal.

So a 1 at position 1 is 128 in decimal, a 1 at position 4 is 16 in decimal, and a 1 at position 8 is 1 in decimal.

We're going to use this table to help us convert decimal numbers into binary.

We're going to move in the table from left to right, starting at position 1, moving through to position 8.

At each position, we're going to follow a process.

It's going to be the same process with a number of rules, and if you learn how to follow these rules, it makes the conversion really easy.

So once again, for a given decimal part of the IP address, so in this case, the first one we're going to look at is 1, 3, 3.

We're going to work through this table left to right, and as I mentioned earlier, I'm going to demonstrate the process for 1, 3, 3 and 33, and then you're going to follow the process for 33 and 7.

So we start on the table in position 1, we compare our decimal numbers, so 133, to the value in the current position of the table, which is 128.

Now, rule number one is that if our number is smaller than the corresponding binary position value, then you write 0 in that position at the table that you're in, in this case, position 1, and then you move on to the next position, starting processing using rule number one again.

However, if our number is greater or equal to the binary position value, then rule 2 applies.

Now, this is the case for us because 133 is more than 128.

So what we do is we minus the binary position value, so 128, from our number, so that means 133 minus 128, this leaves us with 5.

So 5 is the remaining decimal value that we have.

And then we write 1 in this column and move on to position 2.

A binary 1 in this position is equal to 128, so all that we've done is transfer 128 of the decimal value into the corresponding binary value.

So we've added a binary 1 here and we've removed 128 from our decimal number, leaving 5.

And now we can continue the process to convert this remaining 5 into binary.

So now we reset the process, we move to position 2 in the table and start evaluating again using rule number 1.

So we're comparing our remaining value, so we have 5 in decimal, against the binary position value in position 2, which is 64.

So now we compare our number 5 to the value in the table, which is 64.

So in our case, our decimal value of 5 is smaller than the binary value of position 2, and so we're using rule number 1.

We add 0 into this column in the table and then we go right back to the start of the process.

We move on to position 3 and we start evaluating against rule number 1.

We repeat the same process for positions 3, 4 and 5.

Our decimal value of 5 is less than all of those values, so 32, 16 and 8.

So we add 0 in each of those columns and we move on.

So we've evaluated all of those and they match rule number 1 and so we add 0 and we move on.

We're just following the same basic rules.

So now we're at position 6 in the table, we compare our remaining decimal number, so 5, against the value in position 6, which is 4.

Now it's not smaller, so we move past rule 1, it is larger or equal and so we use rule number 2.

We add a value of 1 in this binary position and minus this binary position value from our number, so 5 minus 4 equals 1.

So we have 1 remaining in decimal.

A binary value of 1 in position 6 is equal to 4 in decimal.

So we've just added this to the binary number and removed it from the decimal number.

What this process is doing, bit by bit, is removing value from the decimal value and adding it to the binary 1.

First it was 128 of the decimal number and just now it was 4 of the decimal number.

And next we move on to the next position in the table.

So now we're looking at position 7.

We're comparing our remaining decimal value of 1 with the binary position value.

First we're evaluating against rule 1 and because our remaining value of 1 is less than the binary position value, we use a 0 in this column and we move on to the next position.

So now we're on position 8.

Again we do a comparison.

Our decimal value is 1.

The binary position value is 1 and so we evaluate this against rule 1.

It's not less.

We evaluate it against rule 2.

It is equal or larger.

And so we add a 1 in this column at the table and then we remove the binary position value from our decimal value.

So 1 minus 1 is 0 and because we have 0, the process is finished.

And this means that 133 in decimal is expressed as 1, 0, 0, 0, 1, 0, 1 in binary.

Now this is one example, so this is converting one part of an IP address.

Let's run this process through again with the next part of this IP address.

So this time we're converting the second decimal part of the IP, so 33, into binary.

We start again with our same process and our same table and we move through the table left to right.

We start with position 1.

We compare our decimal number, so 33, to position 1's value, which is 128.

Is our number less than 128?

Yes it is and so we use rule number 1.

We add a 0 into this position at the table and then move on to position 2.

We compare our decimal number to the value in position 2.

Is our number less than 64?

Yes it is and so we use rule number 1.

We add a 0 into the table and move on to position 3.

We compare our decimal number, so 33, to position 3's value, which is 32.

Is our decimal number less than 32?

No it's not, so we skip rule number 1 and move to rule number 2.

We minus 32 from our decimal number 33, leaving us with 1's.

So we've transferred 32 at the decimal value into the binary value and then we move on to position 4 in the table.

Now hope at this point you start to feel more comfortable and we can speed up.

At this point we're at position 4 and we can simply repeat the process.

We compare values if our remaining decimal value is less than the value in the table for that position and we add 0.

And this is the case for positions 4, 5, 6 and 7.

So we add 0's into all of those positions.

Then finally we have position number 8.

And we compare our remaining decimal number, which is 1, with the value in position 8, which is also 1.

Is our value less?

No it's not, so rule number 1 isn't used.

Is our value larger or equal?

Yes it is, and so we use rule number 2.

So we write down 1 in position number 8.

We minus the binary position value, which is 1, from this number, leaving us 0, and that means we've finished the process.

So the binary value for 33 is 0, 0, 1, 0, 0, 0, 0, 1.

Now I want you to try and do this process on your own, on some paper, for the third value of 33, without looking at this lesson.

This process is the same, but it will let you practice yourself.

And once you've reached the same value as me, you can follow the same process for the fourth decimal value of 7.

Once you've done them all, you'll have the full 32-digit binary number, which represents 133.33.33.7.

So go ahead and pause the video and do the final two calculations yourself, and you can resume it once you've finished.

Okay, so I hope that was pretty easy for you, and what you should have for the last decimal value of 7, is a binary value of 0, 0, 0, 0, 0, 1, 1, 1.

If you did, awesome.

That means you've completed it successfully.

If not, don't worry, just watch the first part of this video again, and repeat the process.

Now at this point I want to show you how this works in reverse, converting binary to decimal.

This time let's say that you want to convert this binary IP address into decimal.

So both of these are one and the same.

To start with, we break the IP address into four sections, and this is the same for the binary and the dotted decimal versions.

So each decimal part between the dots represents the corresponding 8-bit binary value.

So I've colored these in red, blue, green, and yellow, so left, middle, left, middle, right, and right.

So this is one and the same.

They're the same IP address expressed in binary on the left and decimal on the right.

Now just as before, the easiest way to tackle this is to work left to right, working on one octet, one by one.

The process of converting this in this direction is much easier.

We have the conversion table just as before.

Each binary bit left to right has a value.

One to eight on the left and one on the right, and we read them one by one.

So we need to the 8-bit binary components, so each of the different colored squares we go through from left to right, and we look whether there's ones or zeros in each position.

If there's a one, we take the corresponding number from the table at the bottom of the screen.

So the first example, the one on the left represents 128.

So we write that down.

We write 128 and then a plus.

If there's a zero, and this is the case for the following four bits, so zero, zero, zero, zero, then we add a zero in our equation.

So we have one to eight, plus zero, plus zero, plus zero, plus zero.

Then we have another one, so we look at the table at the bottom for the corresponding binary position value.

In this case, the number four, so we add that.

Then we have a zero, so we put zero.

And then we have another one, so we again look for the corresponding value in the table, which is a one.

And we add all of those together to get the result.

In this case, 133.

So this represents the first part of the IP address, and we follow the same process for all of the remaining parts.

So zero, zero, one, zero, zero, zero, zero, one represents 33.

We just take each of the ones in each of the components of the IP address and look up the corresponding binary position value in the table.

So in the second component of the IP address, there's a one in position three, which is 32, and a one in position eight, which is one, and this represents 33.

The same is true in the third component, and in the fourth component, there's a one in position six, seven, and eight, which represents four, two, and one.

So we add four, two, and one together to get the result and value.

So if you follow that process bit by bit for each eight-bit component of the binary IP address, then you will end up with the dotted decimal version of the IP address, which is exactly what we've done here.

So why don't you go ahead and pick a random IP address and follow the same process through and see if you get the correct result.

And then once you have the correct result, take that dotted decimal IP address and follow the previous process to convert it from decimal to binary, and you should end up with the same result that you started with.

If you do, that means you understand the end-to-end process of binary to decimal and decimal to binary, and I promise you this does represent a superpower, so it's really important that you understand.

At this point, I'll let everything I want to cover, so go ahead and complete the video, and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this lesson I want to cover a few really important topics which will be super useful as you progress your general IT career, but especially so for anyone who is working with traditional or hybrid networking.

Now I want to start by covering what a VLAN is and why you need them, then talk a little bit about Trump connections and finally cover a more advanced version of VLANs called Q in Q.

Now I've got a lot to cover so let's just jump in and get started straight away.

Let's start with what I've talked about in my technical fundamentals lesson so far.

This is a physical network segment.

It has a total of eight devices, all connected to a single, layer 2 capable device, a switch.

Each LAN, as I talked about before, is a shared broadcast domain.

Any frames which are addressed to all Fs will be broadcast on all ports of the switch and reach all devices.

Now this might be fine with eight devices but it doesn't scale very well.

Every additional device creates yet more broadcast traffic.

Because we're using a switch, each port is a different collision domain and so by using a switch rather than a layer 1 hub we do improve performance.

Now this local network also has three distinct groups of users.

We've got the game testers in orange, we've got sales in blue and finance in green.

Now ideally we want to separate the different groups of devices from one another.

In larger businesses you might have a requirement for different segments of the network from normal devices, for servers and for other infrastructure.

Different segments for security systems and CCTV and maybe different ones for IoT devices and IP telephony.

Now if we only had access to physical networks this would be a challenge.

Let's have a look at why.

Let's say that we talk each of the three groups and split them into either different floors or even different buildings.

On the left finance, in the middle game testers and on the right sales.

Each of these buildings would then have its own switch and the switches in those buildings would be connected to devices also in those buildings.

Which for now is all the finance, all the game tester and all the sales teams and machines.

Now these switches aren't connected and because of that each one is its own broadcast domain.

This would be how things would look in the real world if we only had access to physical networking.

And this is fine if different groups don't need to communicate with us so we don't require cross domain communication.

The issue right now is that none of these switches are connected so the switches have no layer 2 communications between them.

If we wanted to do cross building or cross domain communications then we could connect the switches.

But this creates one larger broadcast domain which moves us back to the architecture on the previous screen.

What's perhaps more of a problem in this entirely physical networking world is what happens if a staff member changes role but not building.

In this case moving from sales to game tester.

In this case you need to physically run a new cable from the middle switch to the building on the right.

If this happens often it doesn't scale very well and that is why some form of virtual local area networking is required.

And that's why VLANs are invaluable.

Let's have a look at how we support VLANs using layer 2 as the OSI 7-Line model.

This is a normal Ethernet frame.

In the context of this lesson what's important is that it has a source and destination MAC address fields together with a payload.

Now the payload carries the data.

The source MAC address is the MAC address of the device which is creating and sending the frame.

The destination MAC address can contain a specific MAC address which means that it's a unique S-frame to a frame that's destined for one other device.

Or it can contain all F's which is known as a broadcast.

And it means that all of the devices on the same layer 2 network will see that frame.

What a standard frame doesn't offer us is any way to isolate devices into different parts, different networks.

And that's where a new standard comes in handy which is known as 802.1Q, also known as .1Q. .1Q changes the frame format of the standard Ethernet frame by adding a new field, a 32-bit field in the middle in Scion.

The maximum size of the frame as a result can be larger to accommodate this new data. 12 bits of this 32-bit field can be used to store values from 0 through to 4095.

This represents a total of 4096 values.

This is used for the VLAN ID or VID.

A 0 in this 12-bit value signifies no VLAN and 1 is generally used to signify the management VLAN.

The others can be used as desired by the local network admin.

What this means is that any .1Q frames can be a member of over 4,000 VLANs.

And this means that you can create separate virtual LANs or VLANs in the same layer 2 physical network.

A broadcast frame so anything that's destined to all PEPs would only reach all the devices which are in the same VLAN.

Essentially, it creates over 4,000 different broadcast domains in the same physical network.

You might have a VLAN for CCTV, a VLAN for servers, a VLAN for game testing, a VLAN for guests and many more.

Anything that you can think of and can architect can be supported from a networking perspective using VLANs.

But I want you to imagine even bigger.

Think about a scenario where you as a business have multiple sites and each site is in a different area of the country.

Now each site has the same set of VLANs.

You could connect them using a dedicated wide area network and carry all of those different company specific VLANs and that would be fine.

But what if you wanted to use a comms provider, a service provider who could provide you with this wide area network capability?

What if the comms provider also uses VLANs to distinguish between their different clients?

Well, you might face a situation where you use VLAN 1337 and another client of the comms provider also uses VLAN 1337.

Now to help with this scenario, another standard comes to the rescue, 802.1AD.

And this is known as Q in Q, also known as provider bridging or stacked VLANs.

This adds another space in the frame for another VLAN field.

So now instead of just the one field for 802.1Q VLANs, now you have two.

You keep the same customer VLAN field and this is known as the C tag or customer tag.

But you add another VLAN field called the service tag or the S tag.

This means that the service provider can use VLANs to isolate their customer traffic while allowing each customer to also use VLANs internally.

As the customer, you can tag frames with your VLANs and then when those frames move onto the service provider network, they can tag with the VLAN ID which represents you as a customer.

Once the frame reaches another of your sites over the service provider network, then the S tag is removed and the frame is passed back to you as a standard .1Q frame with your customer VLAN still tagged.

Q in Q tends to be used for larger, more complex networks and .1Q is used in smaller networks as well as cloud platforms such as AWS.

For the remainder of this lesson, I'm going to focus on .1Q though if you're taking an advanced networking course of mine, I will be returning to the Q in Q topic in much more detail.

For now though, let's move on and look visually at how .1Q works.

This is a cut down version of the previous physical network I talked about, only this time instead of the three groups of devices we have two.

So on the left we have the finance building and on the right we have game testers.

Inside these networks we have switches and connected to these switches are two groups of machines.

These switches have been configured to use 802.1Q and ports have been configured in a very specific way which I'm going to talk about now.

So what makes .1Q really cool is that I've shown these different device types as separate buildings but they don't have to be.

Different groupings of devices can operate on the same layer to switch and I'll show you how that works in a second.

With 802.1Q ports and switches are defined as either access ports or trunk ports and access ports generally has one specific VLAN ID or vid associated with it.

A trunk conceptually has all VLAN IDs associated with it.

So let's say that we allocate the finance team devices to VLAN 20 and the game tester devices to VLAN 10.

We could easily hit any other numbers, remember we have over 4,000 to choose from, but for this example let's keep it simple and keep 10 and 20.

Now right now these buildings are separate broadcast domains because they have separate switches which are not connected and they have devices within them.

Two laptops connected to switch number one for the finance team and two laptops connected to switch number two for the game tester team.

Now I mentioned earlier that we have two types of switch ports in a VLAN cable network.

The first are access ports and the ports which the orange laptops on the right are connected to are examples of access ports.

Access ports communicate with devices using standard Ethernet which means no VLAN tags are applied to the frames.

So in this case the laptop at the top right sends a frame to the switch and let's say that this frame is a broadcast frame.

When the frame exits an access port it's tagged with a VLAN that the access port is assigned to.

In this case VLAN 10 which is the orange VLAN.

Now because this is a broadcast frame the switch now has to decide what to do with the frame and the default behaviour for switches is to forward the broadcast out of all ports except the one that it was received on.

For switches using VLANs this is slightly different.

First it forwards to any other access ports on the same VLAN but the tagging will be removed.

This is important because devices connected to access ports won't always understand 802.1Q so they expect normal untagged frames.

In addition the switch will fold frames over any trunk ports.

A trunk port in this context is a port between two switches for example this one between switch two and switch one.

Now a trunk port is a connection between two dot 1Q capable devices.

It forwards all frames and it includes the VLAN tagging.

So in this case the frame will also be forwarded over to switch one tagged as VLAN 10 which is the gain tester VLAN.

So tagged dot 1Q frames they only get forwarded to other access ports with the same VLAN but they have the tag stripped or they get forwarded across trunk ports with the VLAN tagging intact.

And this is how broadcast frames work.

For unicast ones which go to a specific single MAC address well these will be either forwarded to an access port in the same VLAN that the specific device is on or if the switch isn't aware of the MAC address of that device in the same VLAN then it will do a broadcast.

Now let's say that we have a device on the finance VLAN connected to switch two.

And let's say that the bottom left laptop sends a broadcast frame on the finance VLAN.

Can you see what happens to this frame now?

Well first it will go to any other devices in the same VLAN using access ports meaning the top left laptop and in that case the VLAN tag will be removed.

It will also be forwarded out of any trunk ports tagged with VLAN 20 so the green finance VLAN.

It will arrive at switch two with the VLAN tag still there and then it will be forwarded to any access ports on the same VLAN so VLAN 20 on that switch but the VLAN tagging will be removed.

Using virtual LANs in this way allows you to create multiple virtual LANs or VLANs.

With this visual you have two different networks.

The finance network in green so the two laptops on the left and the one at this middle bottom and then you have the gain testing network so VLAN 10 meaning the orange one on the right.

Both of these are isolated.

Devices cannot communicate between VLANs which are led to networks without a device operating between them such as a layer 3 router.

Both of these virtual networks operate over the top of the physical network and it means that now we can configure this network in using virtual configuration software which can be configured on the switches.

Now VLANs are how certain things within AWS such as public and private vifs on direct connect works so keep this lesson in mind when I'm talking about direct connect.

A few summary points though that I do want to cover before I finish up with this lesson.

First VLANs allow you to create separate layer 2 network segments and these provide isolation so traffic is isolated within these VLANs.

If you don't configure and deploy a router between different VLANs then frames cannot leave that VLAN boundary so they're virtual networks and these are ideal if you want to configure different virtual networks for different customers or if you want to access different networks for example when you're using direct connect to access VPCs.

VLANs offer separate broadcast domains and this is important.

They create completely separate virtual network segments so any broadcast frames within a VLAN won't leave that VLAN boundary.

If you see any mention of 802.1Q then you know that means VLANs.

If you see any mention of VLANs stacking or provide a bridging or 802.1AD or Q in Q this means nested VLANs.

So having a customer tag and a service tag allowing you to have VLANs in VLANs and these are really useful if you want to use VLANs on your internal business network and then use a service provider to provide wide area network connectivity who also uses VLANs and if you are doing any networking exams then you will need to understand Q in Q as well as 802.1Q.

So with that being said that's everything I wanted to cover.

Go ahead and complete this video and when you're ready I'll look forward to you joining me in the next.

Welcome back and in this video I want to step through the architecture and challenges of distributed denial of service attacks known as DDoS attacks.

Now we've got a lot to cover, so let's get started.

Distributed denial of service attacks come in many forms, a few different ways of achieving the same end goal, which is to overload websites or other internet-based services.

The idea is to generate some kind of traffic which competes against legitimate connections and overloads the hardware or software providing the service.

Imagine trying to get into an Apple Store on the day when a new iPhone is released.

How hard is it to get into the store and get service?

What if you added 100,000 random people who just want to queue for no reason and waste the time of the Apple Store staff?

That's the physical equivalent of a DDoS attack.

The challenge when dealing with DDoS attacks comes from the distributed nature of those attacks.

It's hard to identify and block traffic because there can be millions of IP addresses involved with larger internet-scale attacks.

Dealing with DDoS attacks requires specific hardware or software protections.

We won't be covering those in this video, I'm limiting this to just discussing how DDoS attacks work, so the architecture of all the different types of DDoS attacks.

Now DDoS attacks themselves generally fit into one of three categories.

First, application layer attacks such as HTTP floods, and these take advantage of the imbalance of processing between client and server.

It's easy for you to request a web page, but it's often very complex for a server to deliver that same page.

If you multiply that load difference by a billion, it's easy for you to request a web page, but it's often very complex for a server to deliver that same page.

If you multiply that load difference by a billion, then you can have a potentially devastating attack.

Next, we've got protocol-based attacks such as SYNFLUDS, and SYNFLUDS takes advantage of the connection-based nature of requests.

Normally, a connection is initiated via a three-stage handshake, which I detailed in a separate video of this series.

While SYNFLUDS spoof a source ID address and initiate the connection attempt with a server, the server tries to perform step-to-other handshake, but it can't contact the source address because it's spoofed.

In general, it hangs here waiting for a specified duration, and this consumes network resources.

And again, if you multiply this effect by a billion, this can have significant impact on your ability to provide a service.

Lastly, we have volumetric attacks such as DNS amplification, and this relies on how certain protocols such as DNS only take small amounts of data to make the request such as a DNS resolution request, but in response to that, they can deliver a large amount of data.

So one example is an attack of this nature might make requests to DNS servers with a large number of independent requests, where the source address is spoofed to be the actual IP address of our website.

And their servers, potentially hundreds or thousands of them, respond to what they see as legitimate requests and overwhelm a service.

D-dot attacks are often orchestrated by one or a small number of people who are in control of huge botnets, and botnets are constructed of machines such as your laptop or your desktop infected with malware.

Most attacks come from these botnets which are constructed from infected hosts, and the owners of these hosts don't even realize that they're part of the attack.

Now let's look at how these attacks work visually, but before we do that, it's worth reviewing what a valid application architecture should look like.

When a website is working as intended, it looks something like this.

First, we have a number of servers which provide the website functionality, in this case, Categoram.io.

These servers are normally provisioned either based on normal load plus a bit of extra as a buffer, or they're built to autoscale, which means adding more servers when loading creases and removing servers as load decreases.

Now these servers run within a hosting environment which is connected to the public internet via a data connection, which, depending on the speed of this connection, has a limited amount of data that it can transfer and a limit on the number of connections it can handle.

Then our application has users who are using a mobile application to upload their latest captures using TCP port 443.

Now this is HTTPS, and these connections move across our data connection and arrive at the application servers.

Now in normal circumstances, the vast majority of these connections will be from legitimate users of the application.

So this is how it should work.

We have an application, the servers are sized appropriately, we have an appropriate data connection, and our users are accessing the application using this infrastructure.

Now let's step through what happens with the various different forms of DDoT attack.

The first type of DDoT attack is the application layer attack.

Architecturally, behind the scenes we have an attacker who is controlling a network of compromise machines known as a botnet.

This botnet, or more specifically, the machines which form the botnet, are distributed geographically.

[Pause] [Pause] In most circumstances, the real owners of these machines have no knowledge that they've been compromised.

An application-led DDoT attack, as I mentioned at the start of this video, uses the computational imbalance of client-server communications as an attack method.

It's easy, for instance, for the botnets to make simple requests to the application.

In this case, a HTTP GET of a page called reallycomplex.php.

The botnet floats hundreds of thousands, or even tens of thousands of these requests, each of them directed towards the Catergram servers.

This would mean millions or more of these really simple requests, which are all requesting, are reallycomplex.php page.

The issue is that while making these requests is simple, responding to these requests can be computationally expensive, and this can have disastrous effects on the servers.

It's like throwing hand grenades.

They're easy to throw, but they're much more difficult to deal with at the receiving end.

The effect is that our servers, or the data connection, won't have the capacity required to deal with the requests in total.

The fake attack-based requests will prevent the legitimate requests reaching the servers in a timely way, and this can cause performance issues or failures, essentially a general decrease in service levels.

Now, as I mentioned earlier in this video, you can't simply block traffic from individual machines, because there can be millions of them, and the data they're sending can in many ways look exactly the same as legitimate traffic, and this is why you have to handle DDoS attacks in a very specific way.

Now, at this point, let's move on and take a look at another type of DDoS attack.

This time, it's a protocol-based attack.

So, with a protocol-based attack, we follow the same basic architecture, where a single or a small group of attackers is controlling a large botnet, and this botnet is constructed of compromised hosts.

Now, with a protocol attack such as a SYN flood, essentially what happens is a botnet generates a huge number of spoofed SYNs, and SYNs are the initial part of this three-way connection handshake.

So, essentially, all of these individual machines attempt to initiate a connection with the Catergram.io infrastructure, but crucially, they're using a spoofed IP address.

In normal circumstances, if these were real connections, what should happen is our server infrastructure would respond back with SYN acts, which are the second part of the three-way handshake.

Normally, these connection attempts would be from real IP addresses, so IP addresses which are expecting to receive this second part of the three-way handshake.

But because our botnet has initiated these connections with spoofed IP addresses, there won't be anything on the receiving end for our servers to communicate with, and so, in this case, these requests will simply be ignored.

Because they're being ignored, it means the connections will stay in this hung state.

The network resources which would otherwise be used for legitimate connections are waiting for this second part of the three-way handshake, and because the botnet is providing millions and millions of these fake connections, it can mean that network resources are completely consumed with these fake connections, and that means that our legitimate connections won't be able to connect into our infrastructure.

Essentially, by generating this protocol-based attack, this SYN flood, we're preventing the network resources being used for legitimate requests, and so we're essentially significantly impacting the network capability of this application infrastructure.

So, because this three-way handshake is designed to work with slower or less reliable connections, our Catergram.io infrastructure will wait.

It will attempt to connect to these fake source IP addresses.

And while these connections are waiting for the second part of the three-way handshake, these resources can't be used for legitimate connections.

And so, if the botnet is large enough, if it contains a sufficient number of compromised hosts, then it can, in theory, completely take down the service provided by the Catergram.io infrastructure.

Now, let's move on to the third type of attack which I want to cover in this video, and that's known as a volumetric or amplification-based attack.

Now, this type of attack is still orchestrated by a single person or a small group of people, but with this type of attack, the size of the botnet can be much smaller, because an amplification attack exploits a protocol data imbalance.

So, a situation where only a small amount of data is required to initiate a request, but the response to that request is much larger.

In this case, our smaller botnet makes a large number of requests to a large number of DNS servers.

The requests can be made to a large number of DNS servers and be done frequently, because the amount of data that it takes to make the request is relatively small.

Now, the botnet will use a spoofed IP address, and it will use the IP address of our application infrastructure.

So, rather than the DNS servers responding to the botnet, the DNS servers will all respond to our application servers.

Now, the volume of data in each of those responses is much larger than the volume of data making the initial query to DNS.

Because of this, the application servers will be quickly overwhelmed.

This will generally affect the data connection to our application infrastructure, rather than the application server itself, and this will mean that legitimate application users experience degraded levels of performance, because they're competing to use the same total capacity of the application data connection with these fake responses coming in from all of these DNS servers.

So, this type of attack does impact...

Our application's ability to provide service, because the amount of data that our connection provides is consumed, but it's done so in a way which uses amplification.

So, rather than the botnet being required to consume the same amount of bandwidth as our application needs to tolerate, this type of attack can use a tiny amount of bandwidth to initiate the attack, but consume a large amount of bandwidth on the application side, and this makes this type of attack ideally suited to take down larger websites or applications.

So, these are three different common types of DDoS attacks, which you might come across as a solutions architect, an engineer, or a developer.

The important thing to understand about all types of DDoS attack is they can't be combated with normal network protection.

So, because of the distributed nature of the attacks, it's not practical to implement single IP address blocks.

If you're going to block an entire botnet, then you need to block potentially thousands, tens of thousands, hundreds of thousands, or even millions of IP addresses.

What's more, if you use a volumetric or amplification style attack, then the actual machines performing the attack might not even be malicious.

In this case, if you're taking advantage of DNS servers using a DNS amplification attack, then these servers, from their perspective, are doing nothing malicious.

They're just responding to requests.

And so, you have to be really careful that in order to mitigate a DDoS attack, you're not actually blocking legitimate traffic or impacting your application's ability to provide a service.

If you block all DNS servers, then potentially you can have other issues with your application.

Now, AWS and other cloud environments do provide products and services which are specifically designed to help you combat DDoS attacks.

And now that you're aware of the architecture and how these attacks can impact your application, it's going to be much easier for you to understand these different products and services.

Now, with that being said, that's everything that I wanted to cover in this video.

So, go ahead and complete the video.

And when you're ready, I'll look forward to you joining me in the next.

Welcome back.

This is part two of this lesson.

We're going to continue immediately from the end of part one.

So let's get started.

Subnetting is the process of breaking networks up into smaller and smaller pieces.

I've just talked about the class A, B, C, D, and E ranges.

Now, historically, you couldn't break them down.

You were allocated one and that was it.

Classless into domain routing or SIDA, let's just take networks and break them down.

It defines a way of expressing the size of a network.

And this is called a prefix.

An example of this is this /16 network.

So 10.16.0.0/16.

In this case, 16 is the prefix.

Now, you might spot that this is actually inside of the class A network space.

So class A is between 0.anything and 127.anything.

And you might also spot that it's in the private class A address space.

So 10.anything is a private set of addresses.

But this is only actually a subset of this wider network.

10.0.0.0/8 would be the full 10.anything range.

So /8 is the same as a class A network.

The first octet is the network and the rest is available for hosts or subnetting.

This is a /16, which means that the first two octets are the network, so 10.16, and the rest is available for hosts or subnetting.

So this /16 is a smaller network within the 10.0.0.0/8, bigger class A network.

The larger the prefix value, the smaller the network.

And that's a useful one to remember.

Subnetting is a pretty complex process to do well, but you can learn the basics easily enough.

Take this network as an example.

10.16.0.0/16.

If you watched my network fundamental series of videos, you'll know that 10.16.0.0/16 is a range which starts at 10.16.0.0 and ends at 10.16.255.255. /16 tells us that the network part is the first two octets, so 10 and 16.

The network range, therefore, is 0.0 to 255.255 in the hosts part of the IP address.

Now, let's say that we were allocated this range within our organization, but we needed to break it down into multiple networks.

So rather than one large network, let's say, for example, we needed four smaller networks.

But what we can do is to subnet this network.

We can break it down.

All we do is that we break this network into two.

The single /16 becomes two /17 networks.

The first network starts at the starting point of the original network and ends at the halfway point, the point at which the second network starts.

So the first subnetwork is 10.16.0.0 through to 10.16.127.255, so the halfway point.

And the second network starts at 10.16.128.0 and goes through to 10.16.255.255.

So one /16 network is the same as two /17 networks.

But now we can use these two networks for different things within our organization.

Now, we could follow the same process again if we needed more subnets.

For now, we could keep the first /17 network at the top in red, but break the bottom one in green into two networks, so two /18 networks.

The method would be the same, so the first subnetwork, so bottom left, would start at the starting point of the original bottom network, so 10.16.128.0.

The second smaller network, so bottom right, would start at the midway point of the original network.

So these two networks are both /18, which are half the size of the original /17.

And this gives us three subnets, a /17 at the top and two smaller /18s at the bottom.

We could follow the same process again.

Remember, our target here is for subnets, so we can break down the top /17 network into two /18 networks.

The upper left /18 network starts at the starting point of the previous /17 network, and it ends at the halfway point.

The upper right /18 network starts at the midpoint and goes through to the end.

So this is how subnetting and side-up works.

The entire internet is a /0 network.

That's why 0.0.0.0, which you'll see as a default route, matches the entire internet.

All the way through to a /8, which is a class A network, /16, which is a class B network, and /24, which is a class C network.

And then all the way through to /32, which represents a single IP address.

Now this process will become much clearer once you start using this in lab or production environments.

Generally, when you perform subnetting, you'll be breaking down a larger network into two, four, eight, or more smaller networks, always breaking into two and then into two again.

But while it is unusual, it is possible to have odd numbers.

You can break a network into two and then break only one half of that into two more, and this gives you three subnets.

And this is the example at the top right of the screen.

Now this is unusual, but it doesn't break any rules.

Subnetting is the process of taking a larger network and breaking it down into more smaller networks, each of which has a higher prefix, which means a smaller network.

So now that you know the high-level process, I've gone through it graphically.

Let's take a look at this in a little bit more detail before we finish.

We don't use the same example.

And plus before, only now with more detail.

So we start with a /16 network, 10.16.0.0.

Assuming we need four smaller networks, the starting point is to calculate the start and end of this network range.

In this case, 10.16.0.0/16 starts at 10.16.0.0, and finishes at 10.16.255.255.

So we know that any /17 networks will be half of this size.

So step two is to split the original range into two.

The first /17 network starts at the starting point of the original network, so 10.16.0.0, and ends halfway through at the original range, so 10.16.127.255.

So 10.16.0.0/17 means 10.16.0.0 through to 10.16.127.255.

The second smaller network starts at the midpoint, so 10.16.128.0/17, so this starts at 10.16.128.0, and ends at 10.16.255.255.

You've split the original /16 into two.

You've created two smaller /17 networks, each of which occupies half of the original address space.

Now, further splits follow the same process.

Each smaller network has a higher prefix value, and is half the size of the parent's network range.

Its first smaller network starts at the same starting address and finishes halfway, and the second one starts at the halfway point and finishes at the end.

In this case, we have 10.16.128.0/18, and 10.16.192.0/18.

Both of these are within the larger /17 range of 10.16.128.0/17.

If you just think about this process as splitting the network range in half, you're going to create two smaller networks, one which uses the first half and one which uses the second half.

And we can do the same process with the upper subnet, so 10.16.0.0/17, so the network in red.

We can split that range in half, creating two smaller networks.

We've got 10.16.0.0/18, and 10.16.64.0/18.

Now, becoming proficient with this process just takes time.

You need to understand how to calculate IP addresses, how subnet masks and prefixes work, and then you can just follow this process step by step to break down large networks into more and more smaller subnets.

Eventually, you won't even need to calculate it at all.

It will just become instinctive.

I know at this point it might seem like a fair distance off, but I promise it will happen.

Now, at this point, that's everything I wanted to cover in this lesson.

I know it's been a lot of theory.

Go ahead and finish the video.

I'm on your ready.

I look forward to you joining me in the next video.

Welcome back and welcome to another video of this Network Fundamental series where I'll be discussing IP addressing and IP subnetting.

Now we've got a lot to cover, so let's jump in and get started.

IP version 4 addressing has been around since the early days of the internet.

In fact, it was standardized in 1981 via the RFC 791 document, which is attached to this lesson if you want to take a look.

Now it's still the most popular network layer protocol in use on the internet.

IP version 4 addresses occupy a range from 0.0.0.0 to 255.255.255.255.

And this is just under 4.3 billion IP addresses.

Now that sounds like a lot, but with a current world population around the 8 billion mark, that's less than one IP version 4 address per person.

Now the address space was originally fully managed by an organization called IANA, which is the internet assigned numbers authority.

More recently though, parts of the address space have been delegated to regional authorities such as RIPE, ARIN and APNIC.

Now the key thing to understand is that with a few exceptions, IP version 4 addressing is allocated, and that means that you have to be allocated public IP version 4 addresses in order to use them.

You can't simply pick a random address and expect it to work on the public internet without significant issues.

Now there is part of the address space which is private, and that's the addresses which are generally used within home networks, business networks and cloud platforms such as AWS or Azure.

The private address space can be used and reused freely.

So now you know there are 4.294 billion IP version 4 addresses.

You know they start at 0.0.0.0 and end at 255.255.255.255.

Now historically, this range was divided into multiple smaller ranges which are allocated for specific functions.

First, the class A address space which starts at 0.0.0.0 and ends at 127.255.255.255.

Now this range contains 128 networks, each of which has 16.7 million addresses.

So these networks are 0.anything which is reserved 1.anything, 2.anything, all the way up to 127.anything.

The first octet denotes the network with the remaining octets available for hosts or for subnetting as we'll cover later in this video.

So this class of IP addresses, so class A, these were generally used for huge networks and historically these were allocated to huge businesses or organisations which had an internet presence in the early days of the internet.

So businesses like Apple, the Ford Motor Company, the US Postal Service or various parts of the US military.

Many of those organisations have since given up those ranges and these are now allocated to the regional managers of the IP address space for allocation to users in that region.

Now next we have the class B address space and this starts at 128.0.0.0 and it ends at 191.255.255.255.

Now this part of the IP address space offers a total of 16,384 networks, each of them containing 65,536 IP addresses.

So this space was typically used for larger businesses which didn't need a class A allocation.

Like with addresses in the class A space, these are now generally allocated to the regional authorities and they manage them and allocate them out to any organisation who requests and can justify addresses in this range.

Now these networks take the format of 128.0.anything, 128.1.anything, 128.2.anything and then all the way through to 191.253.anything, 191.254.anything and then finally 191.255.anything.

Now with this range of IP addresses so class B, the first two octets are for the network and the last two are for the organisation to assign to devices or to subnet into smaller networks and we'll be talking about that later in this video.

Next we have the class C range which starts at 192.0.0.0 and ends at 223.255.255.255.

Now this range provides over 2 million networks, each containing 256 IP addresses.

So examples of this range include 192.0.1.anything and 192.0.2.anything.

With class C networks, the first three octets denote the network and the remaining is available for hosts or for subnetting.

Class C networks are historically used for smaller businesses who required an IP version 4 presence but weren't large enough for class B or class A addressing and these two are generally now allocated and controlled by regional authorities.

Now there are two more classes, class B and class E, but these are beyond the scope of what this video covers.

Class B is used for multicast and class E is reserved, so I'll cover those at another time.

Now within this public IP version 4 space, certain networks are reserved for private use and this means you can use them however you want, but they aren't roundable across the public IP version 4 internet so these can only be used for private networks or cloud platforms such as AWS who often use them for private networking.

So let's take a look at those.

Private IP addresses are defined within a standards document called RFC1918 and this document defines three ranges of IP version 4 addresses which you're free to use internally.

Now these can't be routed across the internet, but you can use them as you choose internally and as often as required and this is one reason why network address translation is needed to translate these private addresses into publicly roundable addresses so they can communicate with the internet and I cover network address translation in a separate video.

The first private range is a single class A network which starts at 10.0.0.0 and ends at 10.255.255.255 and it provides a total of 16.7 million IP version 4 addresses.

Now this private range is often used within cloud environments and it's generally chopped up into smaller subnetworks which I'll be covering later on in this video.

The next private range is from 172.16.0.0 through to 172.31.255.255.

Now this is a collection of class B networks, 16 of them to be precise so you have 172.16.anything, 172.17.anything, 172.18.anything and so on all the way through to 172.31.anything.

Now each of these networks contains 65,536 addresses and in AWS one of these private ranges 172.31 is used for the default VPC and again these networks are generally broken into smaller subnetworks when used.

Lastly we have 192.168.0.0 to 192.168.255.255 and this range is 256 class C networks so that means 192.168.0.anything, 192.168.1.anything, 192.168.2.anything and so on all the way through to 192.168.255.anything so it provides 256 networks each containing 256 addresses and this range is generally used within home and small office networks so my home network for example uses one of these ranges for all of my devices and my router provides NAT or network address translation services in order to allow them to access the public internet.

Now with any of these ranges you can use them however you want, you can reuse them you can break them up into smaller networks but in all cases you should try and avoid using the same one multiple times.

If you ever need to connect private networks together and they use the same network addressing even if it's private you will have trouble configuring that communication.

Where possible you should always aim to allocate non-overlapping ranges to all of your networks.

So now let's move on to talk about IP version 6 and the differences between it and IP version 4.

So to fully understand the need for IP version 6 and the differences it's useful to start with a representation of the IP version 4 address space.

So we know now that this historically has been broken up into three common classes of IP addresses I've just talked about those.

All of these IP addresses except for a few exceptions are publicly routable meaning if you have one of them configured on a device then you can communicate with another device which is also using a public IP version 4 address.

Now I've also just talked about how part of this IP address space is dedicated for use for private networking but this in its entirety is the IP version 4 address space and in total there are 4,294,967,296 IP addresses so this is the total number of IP version 4 addresses available for use.

Now this might sound like a lot but that's less than one person alive on the planet today and how many of us have a mobile phone and the computer so we have multiple devices already.

What about providers like AWS who have huge public IP addressing requirements?

Well IP version 6 was designed to solve this problem.

The problem that we have far too few IP version 4 addresses and at this point we've essentially exhausted the supply.

With IP version 6 we have more IP addresses to use and to fully appreciate this I want to change the perspective.

This doesn't even do the scale justice but any smaller and you won't be able to see the blue square which now represents the total IP version 4 address space.

Imagine the blue square is actually several thousand times smaller than it is now and with that in mind this is how the IP version 6 address space looks in comparison.

The entire IP version 4 address space available on the public IP version 4 internet is just over 4 billion IP version 4 addresses.

With IP version 6 the entire address space is 340 trillion trillion trillion addresses.

Now humans are bad with large numbers but to put this into perspective it means that there are 670 quadrillion IP version 6 IP addresses per square millimeter of the Earth's surface or to put it another way 50 octillion IP addresses per human alive today or 79 octillion IP version 4 internet's worth of addressing within the IP version 6 address space.

Now think about that for a moment it's enough to give you a headache there are 79 octillion sets of 4.3 billion IP addresses in the IP version 6 address space.

That is an incredibly large number.

Now I don't expect you to remember all of these numbers.

What I want to do is make you comfortable with just how many IP version 6 addresses are available for use.

With IP version 6 the concept of IP addresses as a valuable commodity just goes away.

There are so many of them that you essentially don't require detailed IP management anymore it's just not a scarce resource.

So that's IP version 6.

Next I want to talk about subnetting from an IP version 4 perspective because this is a really useful skill that you should have when you start using a cloud environment.

Okay so this is the end of part one of this lesson.

It was getting a little bit on the long side and so I wanted to add a break.

It's an opportunity just to take a rest or grab a coffee.

Part 2 will be continuing immediately from the end of part one.

So go ahead complete the video and when you're ready join me in part two.

Welcome back, this is part two of this lesson.

We're going to continue immediately from the end of part one, so let's get started.

The principle of Dynamic NAT is similar to static except that devices are not allocated a permanent public IP.

Instead, they're allocated one temporarily from a pool.

Let's say that we have two public IP addresses available for use, 52.95.36.66 and 67.

But we have four devices on the left and all of them at some time need to use public addressing.

So we can't use static NAT because we don't have enough public IP addresses.

With Dynamic NAT, the public to private mapping is allocation-based, so it's allocated as required.

Let's look at an example.

Let's assume that the server on the top left is trying to access the CAT API.

While it creates a packet, the source IP address is itself and the destination IP is the CAT API, which is 1.3.3.7.

So it sends this packet and again the router in the middle is the default gateway for anything which is not local.

As the packet passes through the router or the NAT device, it checks if the private IP has a current allocation of public addressing from the pool and if it doesn't and one is available, it allocates one dynamically and on a temporary basis.

In this case, 52.95.36.67 is allocated on a temporary basis.

So the packets source IP address is translated to this address and the packets are sent onto their final destination.

The CAT API is able to send the response traffic back using this public IP allocation.

So this process is the same so far as if we were using static NAT.

But because Dynamic NAT allocates addressing on a dynamic and temporary basis, multiple private devices can share a single public IP as long as there is no overlap, so as long as the devices use the allocations at different times.

In this case, the upper laptop is accessing the CATFLIX public service using 52.95.36.66 and then afterwards, the lower laptop is using the same public IP address to access the dogflix application.

With Dynamic NAT, because the shared public pool of IP addresses is used, it is possible to run out of public IP addresses to allocate.

If the bottom server attempts to access the public internet, when there are no IPs available in the pool to allocate, then this access will fail.

Now this type of NAT is used when you have less public IPs than private ones, but when all of those private devices at some time need public access, which is bi-directional.

Now the last type of NAT which I want to talk about is the one which you're probably familiar with.

This is port address translation.

This is the type of NAT you likely use on your home network.

Port address translation is what allows a large number of private devices to share one public address.

It's how the AWS NAT gateway functions within the AWS environment.

It has a many to one mapping architecture.

So many private IP version 4 addresses are mapped onto one single public IP version 4 address.

Let's step through an example because this will make it easier to understand.

The example we'll be using is three private devices on the left, all wanting to access Catflix on the right, which has a public IP of 1.3.3.7, and is accessed using TCP port 443, which in this case is HTTPS.

And to make things easier, I'll be colour coding the laptops, so red for the top, purple for the middle, and yellow at the bottom.

Now the way the port address translation or PAT works is to use both IP addresses and ports to allow for multiple devices to share the same public IP.

Every TCP connection, in addition to a source and destination IP address, has a source and destination port.

The destination port for outgoing connections is important because that's what the service runs on.

In this case, Catflix uses the destination port of 443.

The source port, this is randomly assigned by the client.

So as long as the source port is always unique, then many private clients can use the same public IP.

Let's assume that the public IP address at this NAT device is 52.95.36.67.

So at this point, let's say that the top laptop, so the red laptop, generates a packet, and the packet is going to Catflix.

So its destination IP address is 1.3.3.7, and its destination port is 443.

Now the source IP of this packet is itself, so the laptop's private IP address, and the source port is 32.768, which is a randomly assigned ephemeral port.

So this packet is routed through the NAT device on its way to the internet, and in transit, the NAT device records the source IP and the original source private port, and it allocates a new public source port, which in this case is 1.3.3.7.

It records this information inside a NAT table, and it adjusts the packet or translates the packet, so that its source IP address is this single public address which the NAT device uses, and the source port is this newly allocated source port, which is now recorded within the NAT device.

And this newly adjusted packet is forwarded on to Catflix.

If the middle purple laptop did the same thing, then the same process would be followed.

It would record all of this information, it would allocate a new public source port, and it would translate the packet, so adjust the packet's source IP address and the source port, to these newly defined values.

Now if the bottom laptops or the yellow laptop generated a packet, note how this time the source port, which is randomly assigned, is the same source port that the top or red laptop is using for the same connection.

But the same process would be followed.

The NAT device would pick a unique source port to allocate, and it would translate this packet.

It would change the source IP address from the private IP to the single public IP, and it would change the source port of 32.768 to a unique new source port, in this case, 1.3.3.9.

Now normally the reason that only one device can use the same public IP is because these source ports are randomly assigned.

If multiple devices communicate with the same destination service, using the same destination port, and they happen to use the same source port, then it will look like the same connection.

What the NAT device is doing is creating this, a NAT table.

The table is updated with the original private IP and private source port, and the new source IP, which is the public IP address of the NAT device, and then the newly allocated public source port.

This means that when response data comes back, this table can be referenced to ensure that the packet reaches its destination.

So when return traffic occurs, it will be from TCP port 443, with a source IP address of 1.3.3.7.

The destination IP will be the NAT device's public IP, so 52.95.36.67.

And the destination port will be the public source port that NAT device initially translated to.

Let's say in this case, the public source port is 1.3.3.7, which represents the session of the top left laptop.

So for return traffic, if an entry is present in the NAT table, the NAT device translates the public IP and public port, which are the destination IP and port to the original IP, which is 10.0.0.42 to the top laptop, and 32.768, which is the original source port number.

Now it's worth pausing and making sure that you really understand how this process works, because it's how your home route works, and it's how the NAT gateway within AWS works.

Once you understand it, you'll understand why, with port address translation, you can't initiate traffic to these private devices, because without an entry in the NAT table, the NAT device won't know to which device traffic should be translated and forwarded to.

Now, I hope all of this has made sense, and you understand all of the different types of NAT.

NAT is a great topic to understand, because it's one of those things which is used constantly within most architectures, cloud platforms, and even home and business networks.

Now, that's everything I wanted to cover, though, so go ahead and complete this video, and when you're ready, I'll look forward to you joining me in the next video.

Welcome back.

In this lesson, I want to talk about network address translation known as NAT.

NAT is used within home networks, business networks and cloud environments such as AWS.

If you have a thorough understanding of NAT and how it works, it will make using any type of networking including AWS so much easier.

Now I want to keep this video as brief and efficient as possible to let you jump in and take a look at exactly what NAT is and how it works.

So NAT is a process which is designed to address the growing shortage of IP version 4 addresses.

IP version 4 addresses are either publicly routable or they fall within the private address space of IP version 4.

Publicly routable addresses are assigned by a central agency and regional agencies which in turn assign them to ISPs and these ISPs allocate them to business or consumer end users.

An IP version 4 publicly routable addresses have to be unique in order to function correctly.

Private addresses such as those in the 10.0.0.0 range can be used in multiple places but can't be routed over the internet.

And so to give internet access to private devices, we need to use network address translation.

In addition to this, NAT also provides some additional security benefits which I'll be covering soon.

Now there are actually multiple types of NAT which I'm going to cover and all of them, they translate private IP addresses into public IP addresses so the packets can flow over public internet and then translate back in reverse.

So that internet-based hosts can communicate back with these private services.

So that's the high level function of NAT, which each type of NAT handles this process differently.

First we've got static NAT which is where you have a network of private IP version 4 addresses and can allocate a public IP version 4 address to individual private IP addresses.

So the static NAT device translates from one specific private address to one specific public address in effect giving that private address access to the public internet in both directions.

And this is how the internet gateway within AWS works which I'll be covering in another video.

Static NAT is what you would use when certain specific private IP addresses need access to the internet using a public ID and where these IPs need to be consistent.

Dynamic NAT is similar but there isn't this static allocation.

Instead you have a pool of public IP addresses to use and these are allocated as needed so when private IP addresses attempt to use the internet for something.

This method of NAT is generally used when you have a large number of private IP addresses and want them all to have internet access via public IPs but when you have less public IP addresses than private IP addresses and you want to be efficient with how they're used.

Then lastly we have port address translation and this is where many private addresses are translated onto a single public address.

This is likely what your home internet route does, you might have many devices so laptops, computers, tablets, phones and all of those will use port address translation also known as overloading to use a single public IP address.

Now this method as the name suggests uses ports to help identify individual devices and I'll cover in detail how this method works later in this video.

This is actually the method that the NAT gateway or NAT instances use within AWS if you have any AWS experience then you'll recognise this process when I'm talking about the NAT gateway and NAT instances in a separate video.

Now NAT is a process that only makes sense for IP version 4.

Since IP version 6 adds so many more addresses we don't need any form of private addressing and as such we don't need translation.

So try and remember this one IP version 6 generally means you don't need any form of network address translation.

Okay so now I want to step through each of the different methods graphically so you can understand how they work and I'm going to be starting with static network address translation or static NAT.

To illustrate this we want to use a visual example so let's start with a router and NAT gateway in the middle and a private network on the left and then a public network on the right.

We have a situation where we have two devices in the private network, a server and a laptop and both of these need access to external services and let's use the example of Netflix and the CAT API.

So the devices on the left they are private and this means they have addresses in the IP version 4, private address space in this case 10.0.0.10 for the server toward the top and 10.0.0.42 for the laptop toward the bottom.

This means that these two devices packets that they generate cannot be routed over the public internet because they only have private addressing.

Now the CAT API and Netflix both have public IP addresses in the case of the CAT API this is 1.3.7.

So the problem we have with this architecture is that the private addresses can't be routed over the public internet because they're private only.

The public addresses of the public internet-based services can't directly communicate with these private addresses because public and private addresses can't communicate over the public internet.

What we need is to translate the private addresses that these devices have on the left to public IP addresses which can communicate with the services on the right and vice versa.

Now with static NAT the router or NAT device maintains what's known as a NAT table and in the case of static network address translation the NAT table stores a one-to-one device mapping between private IP and public IP.

So any private device which is enabled will have a dedicated, allocated public IP version 4 address.

Now the private device won't have the public IP address configured on it, it's just an allocation.

So let's say that the laptop on the bottom left wants to communicate with Netflix.

Well to do so it generates a packet as normal.

The source IP of the packet is the laptop's private IP address and the destination IP of the packet is one of Netflix's IPs.

Let's say for this example we get issues in DNS.

Now the router in the middle is the default gateway for any destinations so any IP packets which are destined for anything but the local network are sent to this router.

Let's assume that we've allocated a public IP address to this laptop of 52.95.36.67.

So there's an entry in the NAT table containing 10.0.0.42 which is the private address and 52.95.36.67 which is the public address and these are statically mapped to one another.

In this case as the packet passes through the NAT device the source address of the packet is translated from the private address to the applicable public address and this results in this new packet.

So this new packet still has Netflix as the destination but now it has a valid public IP address as the source.

So because we've allocated this bottom laptop a public IP address as the packet moves through the NAT device the NAT device translates the source IP address of this packet from the private laptop's IP address to the allocated public address.

So this is an example of static NAT and for anyone who's interested in AWS this is the process which is performed by the internet gateway so one to one static network address translation.

Now this process works in a similar way in both directions.

So let's say that the API client so the server on the top left wants to communicate with the CAT API.

Well the same process is followed it generates a packet with the destination IP address of the CAT API and it sends it as it's passing through the NAT device the router replaces or translates the source address from the private IP address to the allocated public address.

In this case 52.95.36.68.

The CAT API once it receives the packet sees the source as this public IP so when it responds with data its packet has its IP address as the source and the previous public IP address as the destination the one which is allocated to the server on the top left.

So it sends this packet back to this public IP and remember this public IP is allocated by the NAT device in the middle to the private device at the top left of the API client.

So when this packet arrives at the NAT device the NAT table is checked it sees the allocation is for the server on the top left and so this time for incoming traffic the destination IP address is updated to the corresponding private IP address and then the packet is forwarded through to the private server.

This is how static networks public IPs are allocated to private IPs.

For outgoing traffic the source IP address is translated from the private address to the corresponding public address and for incoming traffic the destination IP address is translated from the allocated public address through to the corresponding private IP address.

Now at no point are the private devices configured with a public IP.

They always have private IP addresses and just to reiterate this is how the AWS internet gateway works which you'll either already know about or will learn about in a different video.

So this is static NAT now let's move on to dynamic NAT.

Okay so this is the end of part one of this lesson.

It was getting a little bit on the long side and so I wanted to add a break.

It's an opportunity just to take a rest or grab a coffee.

Part two will be continuing immediately from the end of part one.

So go ahead complete the video and when you're ready join me in part two.

Welcome back, this is part two of this lesson. We're going to continue immediately from the end of part one. So let's get started.

Now that you know the structure of a segment, let's take a look at how it's used within TCP.

Let's take a few minutes to look at the architecture of TCP.

TCP, like IP, is used to allow communications between two devices.

Let's assume a laptop and a game server.

TCP is connection-based, so it provides a connection architecture between two devices.

And let's refer to these as the client and the server.

Once established, the connection provides what's seen as a reliable communication channel between the client and the server, which is used to exchange data.

Now let's step through how this actually works, now that you understand TCP segments.

The actual communication between client and server, this will still use packets at layer three.

We know now that these are isolated.

They don't really provide error checking, any ordering, and they're isolated, so there's no association between each other.

There's no connection as such.

Because they can be received out of order, and because there are no ports, you can't use them in a situation where there will be multiple applications or multiple clients, because the server has no way of separating what relates to what.

But now we have layer four, so we can create segments.

Layer four takes data provided to it and chops that data up into segments, and these segments are encapsulated into IP packets.

These segments contain a sequence number, which means that the order of segments can be maintained.

If packets arrive out of order, that's okay, because the segments can be reordered.

If a packet is damaged or lost in transit, that's okay, because even though that segment will be lost, it can be retransmitted, and segments will just carry on.

TCP gives you this guaranteed reliable ordered set of segments, and this means that layer four can build on this platform of reliable ordered segments between two devices.

It means that you can create a connection between a client and the server.

In this example, let's assume segments are being exchanged between the client and the game server.

The game communicates to a TCP port 443 on the server.

Now, this might look like this architecturally, so we have a connection from a random port on the client to a well-known port, so 443 on the game server.

So between these two ports, segments are exchanged.

When the client communicates to the server, the source port is 23060, and the destination port is 443.

This architecturally is now a communication channel.

TCP connections are bi-directional, and this means that the server will send data back to the client, and to do this, it just flips the ports which are in use.

So then the source port becomes TCP443 on the server, and the destination port on the client is 23060.

And again, conceptually, you can view this as a channel.

Now, these two channels you can think of as a single connection between the client and the server.

Now, these channels technically aren't real, they're created using segments, so they build upon the concept of this reliable ordered delivery that segments provide, and give you this concept at a stream or a channel between these two devices over which data can be exchanged, but understand that this is really just a collection of segments.

Now, when you communicate with the game server in this example, you use a destination port of 443, and this is known as a well-known port.

It's the port that the server is running on.

Now, as part of creating the connection, you also create a port on your local machine, which is temporary, this is known as the ephemeral port.

This tends to use a higher port range, and it's temporary.

It's used as a source port for any segments that you send from the client to the server.

When the server responds, it uses the well-known port number as the source, and the ephemeral port as the destination.

It reverses the source and destination for any responses.

Now, this is important to understand, because from a layer 4 perspective, you'll have two sets of segments, one with a source port of 23060 and a destination of 443, and ones which are the reverse, so a source port of 443, and a destination of 23060.

From a layer 4 perspective, these are different, and it's why you need two sets of rules on a network ACL within AWS.

One set for the initiating part, so the laptop to the server, and another set for the response part, the server to the laptop.

When you hear the term ephemeral ports or high ports, this means the port range that the client picks as the source port.

Often, you'll need to add firewall rules, allowing all of this range back to the client.

Now, earlier, when I was stepping through TCP segment structure, I mentioned the flags field.

Now, this field contains, as the name suggests, some actual flags, and these are things which can be set to influence the connection.

So, Finn will finish a connection, Akk is an acknowledgement, and Sin is used at the start of connections to synchronize sequence numbers.

With TCP, everything is based on connections.

You can't send data without first creating a connection.

Both sides need to agree on some starting parameters, and this is best illustrated visually.

So, that's what we're going to do.

So, the start of this process is that we have a client and a server.

And as I mentioned a moment ago, before any data can be transferred using TCP, a connection needs to be established, and this uses a three-way handshake.

So, step one is that a client needs to send a segment to the server.

So, this segment contains a random sequence number from the client to the server.

So, this is unique in this direction of travel for segments.

And this sequence number is initially set to a random value known as the ISN or initial sequence number.

So, you can think of this as the client saying to the server, "Hey, let's talk," and setting this initial sequence number.

So, the server receives the segment, and it needs to respond.

So, what it does is it also picks its own random sequence number.

We're going to refer to this as SS, and it picks this as with the client side randomly.

Now, what it wants to do is acknowledge that it's received all of the communications from the client.

So, it takes the client sequence number, received in the previous segment, and it adds one.

And it sets the acknowledgement part of the segment that it's going to send to the CS plus one value.

What this is essentially doing is informing the client that it's received all of the previous transmission, so CS, and it wants it to send the next part of the data, so CS plus one.

So, it's sending this segment back to the client.

It's picking its own server sequence, so SS, and it's incrementing the client sequence by one, and it sends this back to the client.

So, in essence, this is responding with, "Sure, let's talk."

So, this type of segment is known as a SIN-AC.

It's used to synchronize sequence numbers, but also to acknowledge the receipt of the client sequence number.

So, when the first segment was called a SIN, to synchronize sequence numbers, the next segment is called a SIN-AC.

It serves two purposes.

It's also used to synchronize sequence numbers, but also to acknowledge the segment from the client.

The client receives the segment from the server.

It knows the server sequence, and so, to acknowledge to the server that it's received all of that information, it takes the server sequence, so SS, and it adds one to it, and it puts this value as the acknowledgement.

Then it also increments its own client sequence value by one, and puts that as the sequence, and then sends an acknowledgement segment, containing all this information through to the server.

Essentially, it's saying, "Autumn, let's go."

At this point, both the client and server agree on the sequence values.

The client has acknowledged the initial sequence value decided by the server, and the server has acknowledged the initial value decided by the client.

So, both of them are synchronized, and at this point, data can flow over this connection between the client and the server.

Now, from this point on, any time either side sends data, they increment the sequence, and the other side acknowledges the sequence value plus one, and this allows for retransmission when data is lost.

So, this is a process that you need to be comfortable with, so just make sure that you understand every step of this process.

Okay, so let's move on, and another concept which I want to cover is sessions, and the state of sessions.

Now, you've seen this architecture before, a client communicating with the game server.

The game server is running on a well-known port, so TCP 443, and the client is using an ephemeral port 23060 to connect with port 443 on the game server.

So, response traffic will come up from the game server, its source port will be 443, and it will be connecting to the client on destination port 23060.

Now, imagine that you want to add security to the laptop, let's say using a firewall.

The question is, what rules would you add?

What types of traffic would you allow from where and to where in order that this connection will function without any issues?

Now, I'm going to be covering firewalls in more detail in a separate video.

For now though, let's keep this high level.

Now, there are two types of capability levels that you'll encounter from a security perspective.

One of them is called a stateless firewall.

With a stateless firewall, it doesn't understand the state of a connection.

So, when you're looking at a layer 4 connection, you've got the initiating traffic, and you've got the response traffic.

So, the initiating traffic in light with the bottom, and the response traffic in red at the top.

With a stateless firewall, you need two rules.

A rule allowing the outbound segments, and another rule which allows the response segments coming in the reverse direction.

So, this means that the outbound connection from the laptop's IP, using port 23060, connecting to the server IP, using port 443.

So, that's the outgoing part.

And then the inbound response coming from the service IP on port 443, going to the laptop's IP on a femoral port 23060.

So, the stateless firewall, this is two rules, one outbound rule and one inbound rule.

So, this is a situation where we're securing an outbound connection.

So, where the laptop is connecting to the server.

If we were looking to secure, say, a web server, where connections would be made into our server, then the initial traffic would be inbound, and the response would be outbound.

There's always initiating traffic, and then the response traffic.

And you have to understand the directionality to understand what rules you need with a stateless firewall.

So, that's a stateless firewall.

And if you have any AWS experience, that's what a network access control list is.

It's a stateless firewall which needs two rules for each TCP connection, one in both directions.

Now, a stateless firewall is different.

This understands the state of the TCP segment.

So, with this, it sees the initial traffic and the response traffic as one thing.

So, if you allow the initiating connection, then you automatically allow the response.

So, in this case, if we allowed the initial outbound connection from the client laptop to the server, then the response traffic, the inbound traffic, would be automatically allowed.

In AWS, this is how a security group works.

The difference is that a stateless firewall understands level and the state of the traffic.

It's an extension of what a stateless firewall can achieve.

Now, this is one of those topics where there is some debate about whether this is layer four or layer five.

Layer four uses TCP segments and concerns itself with ID addresses and port numbers.

Strictly speaking, the concept of a session or an ongoing communication between two devices, that is layer five.

It doesn't matter if this level item can by layer four and layer five anyway, because it's just easier to explain.

But you need to remember the term stateless and the term stateful and how they change how you create security rules.

For this point, that's everything I wanted to cover.

So, go ahead and complete this video. And when you're ready, I'll look forward to you joining me in the next video of this series.

Welcome back. In this part of the series, I'm going to be primarily covering the transport layer, which is layer 4 of the OSI model.

I'm also going to be touching upon layer 5, which is the session layer of the OSI model, because there is some overlap for certain features, and so it's easier to cover them in one lesson.

The transport layer runs over the top of the network layer and provides most of the functionality, which supports most of the networking, which we use day-to-day on the internet.

The session layer runs on top of the transport layer, and many features, which you might use, are often mixed between these two layers.

Now, as I've already mentioned, it's not generally worth the argument of deciding whether things are covered in layer 4 or layer 5, so I'd only explain both of these layers as one grouping of functionality.

The OSI model is conceptual, after all, and many things exist between or across two different layers.

Now, we've got a lot to cover, so let's jump in and get started.

Before we get started with layer 4, I want to summarize the situation and limitations with layer 3.

Now, we have a functional layer 3, which means that we can communicate between two devices, say a source and destination laptop, using a source and destination IP address.

If both of these use public IP addresses, it doesn't matter where on the internet these devices are, layer 3 and IP routing will ensure that any packets generated and sent from the source laptop will move across any layer 2 networks between the source and destination.

Let's say that using layer 3, the source laptop on the top generates 6 IP packets, and these are all destined for the destination laptop at the bottom right.

The important thing to understand about layer 3 in this context is that each packet is a separate and isolated thing, and it's routed independently over the internet.

It might be logical to assume that the packets arrive in the same state, so the same timing, the same order, and the same quality, but sadly, that's not true.

In ideal conditions, yes, but generally, if you're communicating using only IP, then you're going to have intermittent network conditions, and that can result in a few cases where the arrival condition of packets is different than the condition when they were generated and sent.

One of the first things which we might encounter is out-of-order arrival.

In this case, where packet 3 arrives before packet 2, layer 3, specifically IP, provides no method to ensure the ordering of packet arrival.

For applications which only used IP, this would mean complex logic would need to be built into the application to ensure packets could be sequenced in the same way, and this is not a trivial task.

Because each packet is routed as an independent thing, it's possible packet 2 could have taken a slow, less efficient route, which is why it arrives later.

This is a negative of layer 3, which can be fixed at layer 4.

Another issue with layer 3 is that packets can just go missing.

This can be due to network outages or network conditions, which cause temporary routing loops.

Remember, when I talked about packet structure, I talked about the TTL field, which limited the number of hops a packet could go through.

Well, if the number of hops exceeds this, then it will be discarded.

With IP, there's no reliable method of ensuring packet delivery, and so it's a relatively regular occurrence that packets go missing.

Now, network conditions can also cause delay in delivery, and for any latency-sensitive applications, this can cause significant issues.

The key thing to keep in mind about layer 3, every packet is different.

It's single, it's isolated.

It's a different unit of data which is being routed across a layer 3 network using layer 2 networks as transit.

What happens to one packet might not happen or might happen in a different way to another packet.

Another limitation with layer 3, and this one is probably the one which has the most obvious effect, is that if you think back to the structure of IP packets, they have a source and destination field.

They don't have anything beyond that to distinguish channels of communication.

Packets from a source IP to a destination IP, they're all the same.

You couldn't have two applications running on the source IP, communicating with two applications running on the destination IP, because there's no method of distinguishing between the applications.

Any packet sent by one application would look to be the same as one sent by another.

Think about what you're doing on your device right now.

You might be watching a video.

Do you have a web browser open doing something else?

Do you have an SSH connection or email or any other application which uses the internet?

This means multiple applications, and IP on its own offers no way to separate the packets for individual applications.

This is something which is remedied at layer 4.

Lastly, IP has no flow control.

If a source device is transmitting packets faster than a destination device can receive them, then it can saturate the destination connection and cause loss of data, packets which will be dropped.

Now with only layer 3, we wouldn't have anywhere near the flexibility required to have the internet function in the way that it does.

For that, we need layer 4, and that's what I want to cover in this part of the lesson series.

So what is layer 4 and how does it function?

Let's take a look.

So far, this is what we have network model-wise.

We've discussed the physical layer which is layer 1 at the OSI model.

This relates to how raw bit screen data is transmitted to or received from physical shared media.

We've talked about layer 2 which adds identifiable devices, switches and media access control, but layer 2 ends with isolated layer 2 networks.

In the previous part of this lesson series, I introduced layer 3 which adds IP addressing and routing, so packets can be routed from source to destination across multiple interconnected networks.

Layer 4 builds on top of this.

It adds two new protocols, TCP which stands for transmission control protocol and UDP which stands for user datagram protocol.

Now both of these run on top of IP, and both of them add a collection of features depending on which one of them is used.

Now if you've heard the term TCP/IP, that means TCP running on top of IP.

At a high level, you would pick TCP when you want reliability, error correction and ordering of data.

It's used for most of the important application layer protocols such as HTTP, HTTPS, SSH and so on.

Now TCP is a connection-oriented protocol which means that you need to set up a connection between two devices and once set up, it creates a bidirectional channel of communications.

UDP on your hand is faster because it doesn't have the TCP overhead required for the reliable delivery of data.

This means that it's less reliable.

Now there's a great joke about UDP.

I will tell you about it, but you might not get it.

Anyway, it's a good job my lessons are better than my jokes.

In this lesson, I'm going to spend most of my time talking about TCP because it's used by more of the important protocols that you'll use day-to-day on the internet.

But just know that both TCP and UDP, they both run on top of IP and they're used in the same way.

They use IP as transit.

TCP just offers a more reliable connection-oriented architecture whereas UDP is all about performance.

So there's a simple trade-off.

Now for this lesson series, as I talk about, I'm going to be focusing on TCP because that's what's used for most of the important upper layer protocol.

So let's take a look at exactly how TCP works.

TCP introduces something called segments.

Now a segment is just another container for data like packets and frames before them.

Segments are specific to TCP.

Before we get started talking about the segments themselves, it's important to understand that segments are actually contained in which is known as encapsulated within IP packets.

So let's say that we have a stream of packets.

You know by now that these are all isolated packets.

They're just pieces of data which are routed independently from source to destination.

They're all treated separately.

Well, TCP segments are placed inside packets and the packets carry the segments from their source to their destination.

Segments don't have source or destination IP addresses because they use the IP packets for the transit from source to destination.

This is all handled by layer 3.

In this case, the internet protocol.

TCP segments add additional capabilities to IP packets.

Let's step through the structure of segments so that we can fully understand them.

And I'm going to skip past a few attributes of segments just as I did with layer 3 because there are some parts which are less important or less situational.

So I won't be covering either the options or padding fields within a segment.

The first fields which I want to cover are the source and destination ports.

In addition to the source and destination IP addresses that IP packets provide, TCP segments add source and destination ports.

And this gives the combined TCP/IP protocol the ability to have multiple streams of conversations at the same time between two devices.

When you open the AWS web interface, you're communicating from a port on your local machine to a port on the AWS servers, TCP port 443, which is HTTPS.

Now because of port, you can have multiple streams of communication from your machine.

One to AWS, one to Netflix, and one to this website where you're watching this video.

At the other side, AWS can have multiple streams of communication to their servers.

Each conversation is a unique combination of the source and destination IP, the source port, and the destination port.

All of these four values together identify as a single conversation, a single communications channel.

These two fields ought to allow the internet to function in a flexible way that it does.

It's why SSH and HTTPS can exist on the same EC2 instance and why you can have multiple SSH connections open to the same EC2 instance if you wanted to.

And I'll cover more on how this works as we move through this lesson.

Now next, within the segment, is sequence.

And the sequence number is incremented with each segment that's sent.

And it's unique.

It can be used for error correction if things need to be retransmitted.

You can use to ensure that one IP pass is received and the TCP segments are pulled out.

They can be correctly ordered.

So the sequence number is already uniquely identifying the particular segment within a particular connection so that both sides can make observations about it.

And the way that these observations are done is using app knowledgements.

The app knowledgement field on the right, on the one side, can indicate that it's received up to and including a certain sequence number.

Every segment which is transmitted needs to be acknowledged.

Remember that TCP is a reliable protocol and so if the device is transmitting segment one, two, three, and four to another device, then the other device needs to acknowledge that it's received segment one, two, three, and four.

And this is what the app knowledgement field is for.

So sequence number and app knowledgement are used hand in hand.

Next we have a field called flags and things.

Now within a segment, there is an actual flags component which is nine bits.

And this allows various controls over the TCP segments and the wider connection.

Flags are used to close the connection of the synchronized sequence numbers, but there's also additional things like a data offset and some reserved space.

So I thought this flags and things is essentially the flags plus a number of extra fields which I don't need to go into at this point in the lesson.

Now next we've got the TCP window.

And this is interesting.

This defines the number of bytes that you indicate that you're willing to receive between app knowledgements.

Once reached, the sender will pause until you acknowledge that amount of data.

And this is how flow control is implemented.

It lets the receiver control the rate at which the sender sends data.

If you use a smaller window, it provides additional levels of control over how quickly you're sent data.

Larger windows are more efficient because the header of a TCP segment takes up an amount of space and the smaller the window, the more headers are involved.

So this window setting is quite important if you're using a TCP for practical reasons, but we don't need to go into too much detail in this lesson.

Next we have checks on which is used for error checking.

It means that a TCP layer is able to detect errors and can arrange for retransmission of the data as required.

And then lastly, we have the urgent pointer.

And this is a cool feature.

Imagine if you have a data transfer application where 99% of data is the data being transferred and 1% is control traffic.

So communication between the client and the server, coordinating the actual data transfer.

While setting this field in a segment means that both sides can have separate processing.

So the control traffic always takes priority within the communication.

So any protocols which are latency sensitive and transfer data such as FTP and PellNet can use this field.

Now all of these fields together are known as the TCP header.

And the capacity of a TCP segment which remains is logically enough used for data.

So that's a segment that are placed inside packets and transmitted by one network stack, specifically layer 4 of one network stack and received by another network stack using the layer 4 protocol.

In this case TCP.

Okay so this is the end of part 1 of this lesson.

It was getting a little bit on the long side and so I wanted to add a break.

It's an opportunity just to take a rest or grab a coffee.

Part 2 will be continuing immediately from the end of part 1.

So go ahead, complete video and when you're ready join me in part 2.

Welcome back, this is part three of this lesson. We're going to continue immediately from the end of part two. So let's get started.

The address resolution protocol is used generally when you have a layer three packet and you want to encapsulate it inside a frame and then send that frame to a MAC address.

You don't initially know the MAC address and you need a protocol which can find the MAC address for a given IP address.

For example, if you communicate with AWS, AWS will be the destination of the IP packets.

But you're going to be forwarding via your home router which is the default gateway.

And so you're going to need the MAC address of that default gateway to send the frame to containing the packet.

And this is where ARP comes in.

ARP will give you the MAC address for a given IP address.

So let's step through how it works.

For this example, we're going to keep things simple.

We've got a local network with two laptops, one on the left and one on the right.

And this is a layer three network which means it has a functional layer two and layer one.

What we want is the left laptop which is running a game and it wants to send the packets containing game data to the laptop on the right.

This laptop has an IP address of 133.33.3.10.

So the laptop on the left takes the game data and passes it to its layer three which creates a packet.

This packet has its IP address as the source and the right laptop as the destination.

So 133.33.3.10.

But now we need a way of being able to generate a frame to put that packet in for transmission.

We need the MAC address of the right laptop.

This is what ARP or the address resolution protocol does for us.

It's a process which runs between layer two and layer three.

It's important to point out at this point that now you know how devices can determine if two IP addresses are on the same local network.

In this case, the laptop on the left because it has its subnet mask and IP address as well as the IP address of the laptop on the right.

It knows that they're both on the same network.

And so this is a direct local connection.

Routers aren't required.

We don't need to use any routers for this type of communication.

Now ARP broadcasts on layer two.

It sends an ARP frame to all Fs as a MAC address.

And it's asking who has the IP address 133.33.3.10 which is the IP address of the laptop on the right.

Now the right laptop because it has a full layer one, two and three networks stack is also running the address resolution protocol.

The ARP software sees this broadcast and it responds by saying I'm that IP address.

I'm 133.33.3.10.

Here's my MAC address ending 5B colon 7, 8.

So now the left laptop has the MAC address of the right one.

Now it can use this destination MAC address to build a frame, encapsulate the packet in this frame.

And then once the frame is ready, it can be given to layer one and sent across the physical network to layer one of the right laptop.

Layer one of the right laptop receives this physical orbit stream and hands it off to the layer two software also on the right laptop.

Now it's layer two software reviews the destination MAC address and sees that it's destined for itself.

So it strips off the frame and it sends the packet to its layer three software.

Layer three reviews the packet, sees that it is the intended destination and it de-encapsulates the data.

So strips away the packet and hands the data back to the game.

Now it's critical to understand as you move through this lesson series, even if two devices are communicating using layer three, they're going to be using layer two for local communications.

If the machines are on the same local network, then it will be one layer two frame per packet.

But if you'll see in a moment if the two devices are remote, then you can have many different layer two frames which are used along the way.

And ARP, or the address resolution protocol, is going to be essential to ensure that you can obtain the MAC address for a given IP address.

This is what facilitates the interaction between layer three and layer two.

So now that you know about packets, now that you know about subnet masks, you know about routes and route tables, and you know about the address resolution protocol or ARP, let's bring this all together now and look at a routing example.

So we're going to go into a little bit more detail now.

In this example, we have three different networks.

We've got the orange network on the left, we've got the green network in the middle, and then finally the pink network on the right.

Now between these networks are some routers.

Between the orange and green networks is router one, known as R1, and between the green and pink networks is router two, known as R2.

Each of these routers has a network interface in both of the networks that it touches.

Routers are layer three devices, which means that they understand layer one, layer two, and layer three.

So the network interfaces in each of these networks work at layer one, two, and three.

In addition to this, we have three laptops.

We've got two in the orange network, so device one at the bottom and device two at the top, and then device three in the pink network on the right.

Okay, so what I'm going to do now is to step through two different routing scenarios, and all of this is bringing together all of the individual concepts which I've covered at various different parts of this part of the lesson series.

First, let's have a look at what happens when device one wants to communicate with device two using its IP address.

First, device one is able to use its own IP address and subnet mask together with device two's IP address, and calculate that they're on the same local network.

So in this case, router R1 is not required.

So a packet gets created called P1 with a D2 IP address as the destination.

The address resolution protocol is used to get D2's MAC address, and then that packet is encapsulated in a frame with that MAC address as the destination.

Then that frame is sent to the MAC address of D2.

Once the frame arrives at D2, it checks the frame, hits the destination, so it accepts it and then strips the frame away.

It passes the packet to layer three.

It sees that it's the destination IP address, so it strips the packet away and then passes the game data to the game.

Now all of this should make sense.

This is a simple local network communication.

Now let's step through a remote example.

Device two communicating with device three.

These are on two different networks.

Device two is on the orange network, and device three is on the pink network.

So first, the D2 laptop, it compares its own IP address to the D3 laptop IP address, and it uses its subnet mask to determine that they're on different networks.

Then it creates a packet P2, which has the D3 laptop as its destination IP address.

It wraps this up in a frame called F2, but because D3 is remote, it knows it needs to use the default gateway as a router.

So for the destination MAC address of F2, it uses the address resolution protocol to get the MAC address of the local router R1.

So the packet P2 is addressed to the laptop D3 in the pink network, so the packet's destination IP address is D3.

The frame F2 is now addressed to the router R1 at MAC address, so this frame is sent to router R1.

R1 is going to see that the MAC address is addressed to itself, and so it will strip away the frame F2, leaving just the packet P2.

Now a normal network device such as your laptop or phone, if it received a packet which wasn't destined for it, it would just drop that packet.

A router though, it's different.

The router's job is to route packets, so it's just fine to handle a packet which is addressed somewhere else.

So it reviews the destination of the packet P2, it sees that it's destined for laptop D3, and it has a route for the pink network in its route table.

It knows that for anything destined for the pink network, then router R2 should be the next hop.

So it takes packet P2 and it encapsulates it in a new frame F3.

Now the destination MAC address of this frame is the MAC address of router R2, and it gets this by using the address resolution protocol or ARP.

So it knows that the next hop is the IP address of router R2, and it uses ARP to get the MAC address of router R2, and then it sends this frame off to router R2 as the next hop.

So now we're in a position where router R2 has this frame F3 containing the packet P2 destined for the machine inside the pink network.

So now the router R2 has this frame with the packet inside.

It sees that it's the destination of that frame.

The MAC address on the frame is its MAC address, so it accepts the frame and it removes it from around packet P2.

So now we've just got packet P2 again.

So now router R2 reviews the packet and it sees that it's not the destination, but that doesn't matter because R2 is a router.

It can see that the packet is addressed to something on the same local network, so it doesn't need to worry anymore about routing.

Instead, it uses ARP to get the MAC address of the device with the intended destination IP address, so laptop D3.

It then encapsulates the packet P2 in a new frame, F4, whose destination MAC address is that of laptop D3, and then it sends this frame through to laptop D3. laptop D3 receives the frame, D3 sees that it is the intended destination of the frame because the MAC address matches its MAC address.

It strips off the frame, it also sees that it's the intended destination of the IP packet, it strips off the packet, and then the data inside the packet is available for the game that's running on this laptop.

So it's a router's job to move packets between networks.

Router's doing this by reviewing packets, checking route tables for the next hop or target addresses, and then adding frames to allow the packets to pass through intermediate layer 2 networks.

A packet during its life might move through any number of layer 2 networks and be re-encapsulated many times during its trip, but normally the packet itself remains unchanged all the way from source to destination.

A router is just a device which understands physical networking, it understands data link networking, and it understands IP networking.

So that's layer 3, the network layer, and let's review what we've learned quickly before we move on to the next layer of the OSI model.

Now this is just an opportunity to summarize what we've learned, so at the start of this video, at layer 2 we had media access control, and we had device to device or device to all device communications, but only within the same layer 2 network.

So what does layer 3 add to this?

Well it adds IP addresses, either version 4 or version 6, and this is cross network addressing.

It also adds the address, resolution, protocol, or ARP, which can find the MAC address for this IP address or for a given IP address.

Layer 3 adds routes, which define where to forward a packet to, and it adds route tables, which contain multiple routes.

It adds the concept of a device called a router, which moves packets from source to destination, encapsulating these packets in different layer 2 frames along the way.

This altogether allows for device to device communication over the internet, so you can access this video, which is stored on a server, which has several intermediate networks away from your location.

So you can access this server, which has an IP address, and packets can move from the server through to your local device, crossing many different layer 2 networks.

Now what IP doesn't provide?

It provides no method of individual channels of communication.

Layer 3 provides packets, and packets only have source IP and destination IP, so for a given two devices, you can only have one stream of communication, so you can't have different applications on those devices communicating at the same time.

And this is a critical limitation, which is resolved by layers 4 and above.

Another element of layer 3 is that in theory packets could be delivered out of order.

Individual packets move across the internet through intermediate networks, and depending on network conditions, there's no guarantee that those packets will take the same route from source to destination, and because of different network conditions, it's possible they could arrive in a different order.

And so if you've got an application which relies on the same ordering at the point of receipt as at the point of transmission, then we need to add additional things on top of layer 3, and that's something that layer 4 protocols can assist with.

Now at this point we've covered everything that we need to for layer 3.

There are a number of related subjects which I'm going to cover in dedicated videos, such as network address translation, and how the IP address space functions, as well as IP version 6, which in this component of the lesson series, we've covered how the architecture of layer 3 of the OSI model works.

So at this point, go ahead and complete this video, and then when you're ready, I'll look forward to you joining me in the next part of this lesson series where we're going to look at layer 4.

Welcome back, this is part two of this lesson.

We're going to continue immediately from the end of part one, so let's get started.

Now we talked about the source and destination IP address of these packets, so now let's focus on IP addressing itself.

IP addressing is what identifies a device which uses layer 3 IP networking.

Now I'll talk more about how IP addressing is decided upon and assigned in another video, for now I want you to fully understand the structure of an IP address.

In this video I'll be focusing on IP version 4, because I have a separate video which will cover IP version 6 in depth.

This is an IP address, 133.33.3.7.

From a pure network connectivity point of view, if you have a valid IP version 4 address, you can send packets to 133.33.3.7 and they will at least start on the journey of getting to this destination.

Now there might be blocks in the way, so firewalls or other security restrictions, all the IP could be offline, but packets will move from you over the internet on their way to this IP address.

Now this format is known as dotted decimal notation.

It's four decimal numbers from 0 to 255 separated by dots.

So 133.33.3.7.

Now all IP addresses are actually formed of two different parts.

There's the network part which states which IP network this IP address belongs to, and then the host part which represents hosts on that network.

So in this example the network is 133.33, and then the hosts on that network can use the remaining part of the IP.

In this case 3.7 is one device on that network, a laptop.

A really important part of understanding how your data gets from your location to a remote network is the given two IP addresses.

How do you tell if they're on the same IP network or different IP networks?

If the network part of the IP address matches between two different IP addresses, then they're on the same IP network.

If not, they're on different IP networks.

So you need to be able to calculate where you've an IP address, which part of that address is the network, and which part is the host.

And by the end of this lesson you will know how to do that.

Now IP addresses are not actually dotted decimal.

That's how they're represented for humans.

They're actually binary numbers.

Each decimal part of the IP address is an 8-bit binary number.

There are four of these per IP version 4 address, and this means that an entire IP address is 32 bits in size.

So four sets of 8 bits, and each of these 8 bits is known as an octet.

You might hear somebody refer to say the first and second octet of an IP address, and this is always read left to right.

The first octet in this example is 1, 3, 3, or in binary 1, 0, 0, 0, 1, 0, 1.

And the second octet is 33, which in binary is 0, 0, 1, 0, 0, 0, 0, 1.

Now this binary conversion, this is not something which I'm going to cover in this lesson, but I will make sure there's a link attached to the lesson which shows you how to do it.

It's just decimal to binary maths, and once you know how it's done, it's really easy to do, even in your head.

Now I'm going to talk about how you can determine which IPs are on the same network next, but I wanted to introduce the format of IP addresses first.

In this example, this IP address has what's known as a /16 prefix.

This means that the first 16 bits represent the network, and the rest are for hosts.

Now I don't really talk about how this works in detail coming up next.

Because the first 16 bits are network, it means that the second IP address is 1, 3, 3, .33, .33, .37, because the network part of that matches is 1, 3, 3, .33, and it's on the same IP network.

I'm going to detail coming up next how this calculation is done.

For now, I want you to be comfortable knowing that if the network component of two IP addresses match, then devices are local.

If they don't match, then devices are remote.

That matters when we start covering IP routing.

Now IP addresses are networks.

These are either statically assigned by humans, and this is known as a static IP, or they're assigned automatically by machines.

So service on your network running DHCP service software.

Now DHCP stands for Dynamic Host Configuration Protocol, and this is something I'll be covering in detail in a separate video.

On a network, IP addresses need to be unique, or bad things happen.

Globally, in most cases, IP addresses need to be unique, or also bad things happen.

So keep that in mind.

Generally, when you're dealing with IP addresses, you want them to be unique, especially on your local network.

Now let's talk about subnet masks, because these are what helps us determine if IP addresses are local to each other or remote.

Subnet masks are a critical part of IP networking.

They're configured on layer 3 interfaces, along with IP addresses.

What's also configured on most network interfaces is a default gateway.

This is an IP address on a local network, which packets are forwarded to, generally, if the intended destination is not a local IP address.

Subnet masks are what allow an IP device to know if an IP address which it's communicating with is on the same network or not, and that influences if the device attempts to communicate directly on the local network, or if it needs to use the default gateway.

On your home network, for example, your internet router is likely set as your default gateway, so when you browse to Netflix.com or interact with AWS because the IP addresses that you're talking to are not local, then packets from your machine are passed to your router, which is the default gateway.

So let's say that we have an IP address, 133.33.3.7.

Now this alone is just a single address.

We don't know which part of it is the network and which part of it is the host component.

I just finished talking about how IP addresses can match binary numbers.

This IP address in binary is 1-0-0-0-1-0-1, so that's the first octet, and then 0-0-1-0-0-1, that's the second octet, and then 0-0-0-0-0-0-1-1, that's the third octet, and then finally 0-0-0-0-0-1-1-1, and that's the fourth octet, and that represents 133.33.3.7.

So as a reminder, if we're dealing manually with subnet masks, and remember this is something that's generally performed in software by your networking stack, the first thing we need to do is convert the dotted decimal notation into a binary number.

Now along with this IP address, we would generally also configure either statically or using DHCP, a subnet mask.

In this example, the subnet mask that we have is 255.255.0.0 or /16, and these mean the same thing, and I'll show you why over the next few minutes.

A subnet mask represents which part of the IP is for the network.

It helps you, or more often a machine, know which part of an IP address is which.

To use a subnet mask, you first have to convert it into binary, so 255.255.0.0 is this in binary.

We convert it just like an IP address.

So the first octet is all 1s, the second octet is all 1s, the third and fourth octet are all 0s.

The /16, which is known as the prefix, this is just shorthand.

It's the number of 1s in the subnet mask starting from the left.

So /16 simply means 16 1s, which is the same as 255.255.0.0 when you convert that into binary.

Now when you have the subnet mask in binary, anything with a 1 represents the network, anything with a 0 represents the host component.

So if you overlay a subnet mask and an IP address, both of them in binary, it becomes really easy to tell which part is which.

Something else which is really cool is that for a given network, you can calculate the start and end IP addresses of that network.

Take for example, the IP address that's on screen now, so 133.33.3.7.

Well we've converted that into binary and we've also converted the subnet mask of 255.255.0.0 also into binary.

So that's in blue, right below the binary IP address.

To calculate the start of the network, we begin with the network part of the IP address and then for the host part, we have all 0s.

So let's look at what we've done.

The subnet mask, where there are 1s, this is the network part.

So we take the original IP address and where the subnet mask has 1s, that's the network part, so 133.33.

Then for the part which is hosts, which is where the subnet mask shows 0s, then we have all 0s.

This means that the network starting point is 133.33.0.0.

Now to find the end, we take the network component of the IP address again, so where the subnet mask is all 1s, that's what we start with.

And to work out the end of the network, we take the host component, so where the subnet mask is 0s, and we have all 1s in the IP address.

So the ending part of this network is 133.33.255.255.

So the starting address of a network is the network component of the IP address, identified with the subnet mask, and then all 0s for the host part of the IP address, and the ending address of the network is the network part of the IP address to start with, and then for the host component, we have all 1s.

So this is how subnet masks work.

They're used to identify which part of an IP address is the network part and which is the host part.

As long as the network part for two different IP addresses is the same, then we know that both of those IP addresses are on the same IP network, and this is essential so that the machine can identify when it can send data directly on the same local network, or when IP routing needs to be used to transfer packets across different intermediate networks.

So it's how your local device, your local laptop, knows to send packets to your internet router for Netflix or AWS, rather than trying to look for both of those systems locally on your local area network.

And that's how a router makes that decision too, when it's looking where to forward packets to.

So using subnet masks and IP addresses, it's how a lot of the intelligence of layer 3 is used.

Now next, I want to spend some time looking at route tables and routes.

Let's step through an example of data moving from you to AWS, and I want to keep focus for now on how a router makes a decision where to send data.

Packets that you create for AWS will move from your house into your internet provider across the internet, potentially even between countries, and then finally arrive at AWS.

Let's step through a simple example.

So we start with our house on the left.

Next, we have our internet provider known as an ISP or Internet Service Provider, and let's call this Meow ISP, and then we have three destination networks.

We have AWS, our ISP's upstream provider, and then Netflix.

Now we want to communicate with AWS, and so we create a packet on our local device, which has our IP address 1.3.3.7 as the source IP address, and it has a destination IP address of 52.217.13.37.

Now you're going to have an internet router within your home, and this is where your device will send all of its data through.

That router has what's known as a default route, which means all IP traffic is sent to it on its way to Meow ISP.

Now I'll explain what a default route is in a second.

For now, just assume that all data that you generate within your local network by default is sent through to your internet service provider.

So now the packet that you've generated is inside your internet service provider on a router, and this router has multiple network interface cards connecting to all of those remote networks.

Now let's assume in those remote networks is another router, and each of these routers uses the dot 1 IP address in each of those networks.

So how does the ISP router inside Meow ISP know where to forward your data to?

Well, it uses routes and route tables.

Every router will have at least one route table.

It could have more, which are attached to individual network interfaces, but for now let's keep things simple and assume that the router within our ISP has a single route table, and it will look something like this.

A route table is a collection of routes.

Each row in this table is an example route.

It will have a destination field, and it will have a next hop or a target field.

What happens is that every packet which arrives at this router, the router will check the packet's destination.

What IP address is this packet destined for?

And in this example, it's 52.217.13.37.

Now at this point, the router will look for any routes in the route table which match the destination IP address of this packet.

If multiple routes match, then it will prefer ones which are more specific.

The two routes in yellows at the top and the bottom, these are examples of fairly specific routes.

The one in blue in the middle is the inverse, this is not a specific route.

The larger the prefix, so the higher the number after the slash, the more specific the route.

So a slash 32 is the most specific, and a slash 0 is the least specific.

A slash 32 actually represents one single IP address, and a slash 0, well this represents all IP addresses.

A slash 24 means that the first 24 bits are for the network, and the last 8 bits are for the host.

So this matches a network of 256 IP addresses.

So for this packet that we have with the destination of 52.217.13.37, we've got two routes which match.

The top route, which is 52.217.13.0/24, that network contains the IP address which our packet is destined for.

So this matches.

But also the middle route, 0.0.0/0, this matches, because this matches all IP addresses.

The middle route is known as a default route.

I mentioned before the packets from our home network on the left arrive at our ISP because there's a default route.

Well this 0.0.0/0 is an example of a default route.

This will match if nothing else does.

Because we have two more specific routes in this route table, so the top and bottom, if either of those match, they will be selected rather than the default route in the middle.

In this case the bottom route doesn't match our particular packet, only the top one matches.

And so the top route will be selected because it's more specific than the default route.

Now for the route that's selected, so the top route, it has a next hop or target field.

This is the IP address which the packet is going to be forwarded to, to get one step closer through to its destination.

Or in this case to arrive at the actual destination.

And so the packet is forwarded through to this address.

Routing as a process is where packets are forwarded or routed hop by hop across the internet from source to destination.

Route tables are the thing which enables this.

Route tables can be statically populated, or there are protocols such as BGP or the border gateway protocol, which allow routers to communicate with each other to exchange which networks they know about.

And this is how the core of the internet functions.

One important thing that you need to understand though, is that when our ISP router is forwarding the packet through to the AWS router, it's forwarding it at layer 2.

It wraps the packet in a frame.

The packet doesn't change.

The frame though, it has the AWS routers MAC address as its destination.

That's how the packet gets to the AWS router.

But how do we determine the MAC address of the AWS router in this example?

For that we use something called the address resolution protocol, and that's what I'm going to be covering next.

This is the end of part 2 of this lesson.

It's a pretty complex lesson, and so I wanted to give you the opportunity to take a small break, maybe stretch your legs, or make another coffee.

Part 3 will continue immediately from this point, so go ahead, complete this video, and when you're ready, I look forward to you joining me in part 3.

Welcome back.

Now that we've covered the physical and data link layers, next we need to step through layer 3 of the OSI model, which is the network layer.

As I mentioned in previous videos, each layer of the OSI model builds on the layers below it, so layer 3 requires one or more operational layer 2 networks to function.

The job of layer 3 is to get data from one location to another.

When you're watching this video, data is being moved from the server hosting the video through to your local device.

When you access AWS or stream from Netflix, data is being moved across the internet, and it's layer 3 which handles this process of moving data from a source to a destination.

To appreciate layer 3 fully, you have to understand why it's needed.

So far in the series, I've used the example of 2-4 friends playing the game on a local area network.

Now what if we extended this, so now we have 2 local area networks and they're located with some geographic separation.

Let's say that one is on the east coast of the US and another is on the west coast, so there's a lot of distance between these 2 separate layer 2 networks.

Now LAN1 and LAN2 are isolated layer 2 networks at this point.

Devices on each local network can communicate with each other, but not outside of that local layer 2 network.

Now you could pay for and provision a point-to-point link across the entire US to connect these 2 networks, but that would be expensive, and if every business who had multiple offices needed to use point-to-point links, it would be a huge mess and wouldn't be scalable.

Additionally, each layer 2 network uses a shared layer 2 protocol.

In the example so far, this has been Ethernet.

Any networks where only using layer 2, if we want them to communicate with each other, they need to use the same layer 2 protocol to communicate with another layer 2 network.

Now not everything uses the same layer 2 protocol, this presents challenges, because you can't simply join 2 layer 2 networks together, which use different layer 2 protocols and have them work out of the box.

With the example which is on screen now, imagine if we had additional locations spread across the continental US.

Now in between these locations, let's add some point-to-point links, so we've got links in pink which are tabled connections, and these go between these different locations.

Now we also might have point-to-point links which use a different layer 2 protocol.

In this example, let's say that we had a satellite connection between 2 of these locations.

This is in blue, and this is a different layer 2 technology.

Now Ethernet is one layer 2 technology which is generally used for local networks.

It's the most popular wired connection technology for local area networks.

But for point-to-point links and other long distance connections, you might also use things such as PPP, MPLS or ATM.

Not all of these use frames with the same format, so we need something in common between them.

Layer 2 is the layer of the OSI stack which moves frames, it moves frames from a local source to a local destination.

So to move data between different local networks, which is known as inter-networking, this is where the name internet comes from.

We need a layer 3.

Layer 3 is this common protocol which can span multiple different layer 2 networks.

Now layer 3 or the network layer can be added onto one or more layer 2 networks, and it adds a few capabilities.

It adds the internet protocol or IP.

You get IP addresses which are cross-networking addresses, which you can assign to devices, and these can be used to communicate across networks using routing.

So the device that you're using right now, it has an IP address.

The server which stores this video, it too has an IP address.

And the internet protocol is being used to send requests from your local network across the internet to the server hosting this video, and then back again.

IP packets are moved from source to destination across the internet through many intermediate networks.

Devices called routers, which are layer 3 devices, move packets of data across different networks.

They encapsulate a packet inside of an ethernet frame for that part of the journey over that local network.

Now encapsulation just means that an IP packet is put inside an ethernet frame for that part of the journey.

Then when it needs to be moved into a new network, that particular frame is removed, and a new one is added around the same packet, and it's moved onto the next local network.

So as this video data is moving from my server to you, it's been wrapped up in frames.

Those frames are stripped away, new frames are added, all while the packets of IP data move from my video server to you.

So that's why IP is needed at a high level, to allow you to connect to all that remote networks, crossing intermediate networks on the way.

Now over the coming lesson, I want to explain the various important parts of how layer 3 works.

Specifically IP, which is the layer 3 protocol used on the internet.

Now I'm going to start with the structure of packets, which are the data units used within the internet protocol, which is a layer 3 protocol.

So let's take a look at that next.

Now packets in many ways are similar to frames.

It's the same basic concept.

They contain some data to be moved, and they have a source and destination address.

The difference is that with frames, both the source and destination are generally local.

With an IP packet, the destination and source addresses could be on opposite sides of the planet.

During their journey from source to destination packets remain the same, as they move across layer 2 networks.

They're placed inside frames, which is known as encapsulation.

The frame is specific to the local network that the packet is moving through, and changes every time the packet moves between networks.

The packet though doesn't change.

Normally it's constant for the duration for its entire trip between source and destination.

Although there are some exceptions that I'll be detailing in a different lesson, when I talk about things like network address translation.

Now there are two versions of the internet protocol in use.

Version 4, which has been used for decades, and version 6, which adds more scalability.

And I'll be covering version 6 and its differences in a separate lesson.

An IP packet contains various different fields, much like frames that we discussed in an earlier video.

At this level there are a few important things within an IP packet which you need to understand, and some which are less important.

Now let's just skip past the less relevant ones.

I'm not saying any of these are unimportant, but you don't need to know exactly what they do at this introductory level.

Things which are important though, every packet has a source and destination IP address field.

The source IP address is generally the device IP which generates the packet, and the destination IP address is the intended destination IP for the packet.

In the previous example we have two networks, one east coast and one west coast.

The source might be a west coast PC, and the destination might be a laptop within the east coast network.

But crucially these are both IP addresses.

There's also the protocol field, and this is important because IP is layer 3.

It generally contains data provided by another layer, a layer 4 protocol, and it's this field which stores which protocol is used.

So examples of protocols which this might reference are things like ICMP, TCP or UDP.

If you're storing TCP data inside a packet this value will be 6, for PINs known as ICMP this value will be 1, and if you're using UDP as a layer 4 protocol then this value will be 17.

This field means that the network stack at the destination, specifically the layer 3 component of that stack, will know which layer 4 protocol to pass the data into.

Now the bulk of the space within a packet is taken up with the data itself, something that's generally provided from a layer 4 protocol.

Now lastly there's a field called time to live or TTL.

Remember the packets will move through many different intermediate networks between the source and the destination, and this is a value which defines how many hops the packet can move through.

It's used to stop packets looping around forever.

If for some reason they can't reach their destination then this defines a maximum number of hops that the packet can take before being discarded.

So just in summary a packet contains some data which it carries generally for layer 4 protocols.

It has a source and destination IP address, the IP protocol implementation which is on routers moves packets between all the networks from source to destination, and it's these fields which are used to perform that process.

As packets move through each intermediate layer 2 network, it will be inserted or encapsulated in a layer 2 frame, specific for that network.

A single packet might exist inside tens of different frames throughout its route to its destination, one for every layer 2 network or layer 2 point to point link which it moves through.

Now IP version 6 from a packet structure is very similar, we also have some fields which matter less at this stage.

They are functional but to understand things at this level it's not essential to talk about these particular fields.

And just as with IP version 4, IP version 6 packets also have both source and destination IP address fields.

But these are bigger IP version 6 addresses are bigger which means there are more possible IP version 6 addresses.

And I'm going to be covering IP version 6 in detail in another lesson.

It means though that space taken in a packet to store IP version 6 source and destination addresses is larger.

Now you still have data within an IP version 6 packet and this is also generally from a layer 4 protocol.

Now strictly speaking if this were to scale then this would be off the bottom of the screen, but let's just keep things simple.

We also have a similar field to the time to live value within IP version 4 packets, which in IP version 6 this is called the hop limit.

Functionally these are similar, it controls the maximum number of hops that the packet can go through before being discarded.

So these are IP packets, generally they store data from layer 4 and they themselves are stored in one or more layer 2 frames as they move around networks or links which fall on the internet.

Okay so this is the end of part 1 of this lesson.

It was getting a little bit on the long side and I wanted to give you the opportunity to take a small break, maybe stretch your legs or make a coffee.

Now part 2 will continue immediately from this point, so go ahead complete this video and when you're ready I look forward to you joining me in part 2.

Welcome back and in this part of the lesson series I'm going to be discussing layer one of the seven layer OSI model which is the physical layer.

Imagine a situation where you have two devices in your home let's say two laptops and you want to play a local area network or LAN game between those two laptops.

To do this you would either connect them both to the same Wi-Fi network or you'd use a physical networking cable and to keep things simple in this lesson I'm going to use the example of a physical connection between these two laptops so both laptops have a network interface card and they're connected using a network cable.

Now for this part of the lesson series we're just going to focus on layer one which is the physical layer.

So what does connecting this network cable to both of these devices give us?

Well we're going to assume it's a copper network cable so it gives us a point-to-point electrical shared medium between these two devices so it's a piece of cable that can be used to transmit electrical signals between these two network interface cards.

Now physical medium can be copper in which case it uses electrical signals it can be fiber in which case it uses light or it can be Wi-Fi in which case it uses radio frequencies.

Whatever type of medium is used it needs a way of being able to carry unstructured information and so we define layer one or physical layer standards which are also known as specifications and these define how to transmit and receive raw bitstream so ones and zeros between a device and a shared physical medium in this case the piece of copper networking cable between our two laptops so the standard defines things like voltage levels, timings, data rates, distances which can be used, the method of modulation and even the connector type on each end of the physical cable.

The specification means that both laptops have a shared understanding of the physical medium so the cable.

Both can use this physical medium to send and receive raw data.

For copper cable electrical signals are used so a certain voltage is defined as binary 1 say 1 volt and a certain voltage as binary 0 say -1 volt.

If both network cards in both laptops agree because they use the same standard then it means that zeros and ones can be transmitted onto the medium by the left laptop and received from the medium by the right laptop and this is how two networking devices or more specifically two network interface cards can communicate at layer one.

If I refer to a device as layer X so for example layer one or layer three then it means that the device contains functionality for that layer and below so a layer one device just understands layer one and a layer three device has layers one, two and three capability.

Now try to remember that because it's going to make much of what's coming over the remaining videos of this series much easier to understand.

So just to reiterate what we know to this point we've taken two laptops we've got two layer one network interfaces and we've connected them using a copper cable a copper shared medium and because we're using a layer one standard it means that both of these cards can understand the specific way that binary zeros and ones are transmitted onto the shared medium.

Now on the previous screen I use the example of two devices so two laptops with network interface cards communicating with each other.

Two devices can use a point-to-point layer one link a fancy way of talking about a network cable but what if we need to add more devices a two-player game isn't satisfactory we need to add two more players for a total of four.

Well we can't really connect these four devices to a network cable with only two connectors but what we can do is to add a networking device called a hub in this example it's a four-port hub and the laptop on the left and right instead of being connected to each other directly and now connected to two ports of that hub because it's a four-port hub this also means that it has two ports free and so it can accommodate the top and bottom laptops.

Now hubs have one job anything which the hub receives on any of its ports is retransmitted to all of the other ports including any errors or collisions.

Conceptually a hub creates a four connector network cable one single piece of physical medium which four devices can be connected to.

Now there are a few things that you really need to understand at this stage about layer one networking.

First there are no individual device addresses at layer one one laptop cannot address traffic directly at another it's a broadcast medium the network card on the device on the left transmits onto the physical medium and everything else receives it it's like shouting into a room with three other people and not using any names.

Now this is a limitation but it is fixed by layer two which will cover soon in this lesson series.

The other consideration is that it is possible that two devices might try and transmit at once and if that happens there will be a collision this corrupts any transmissions on the shared medium only one thing can transmit at once on a shared medium and be legible to everything else if multiple things transmit on the same layer one physical medium then collisions occur and render all of the information useless.

Now related to this layer one has no media access control so no method of controlling which devices can transmit so if you decide to use a layer one architecture so a hub and all of the devices which is shown on screen now then collisions are almost guaranteed and the likelihood increases the more layer one devices are present on the same layer one network.

Layer one is also not able to detect when collisions occur remember these network cards are just transmitting via voltage changes on the shared medium it's not digital they can in theory all transmit at the same time and physically that's okay it means that nobody will be able to understand anything but at layer one it can happen so layer one is done it doesn't have any intelligence beyond defining the standards that all of the devices will use to transmit onto the shared medium and receive from the shared medium because of how layer one works and because of how a hub works because it simply retransmits everything even collisions then the layer one network is said to have one broadcast and one collision domain and this means that layer one networks tend not to scale very well the more devices are added to a layer one network the higher the chance of collisions and data corruption.

Now layer one is fundamental to networking because it's how devices actually communicate at a physical level but for layer one to be useful for it to be able to be used practically for anything else then we need to add layer two and layer two runs over the top of a working layer one connection and that's what we'll be looking at in the next part of this lesson series.

As a summary of the position that we're in right now assuming that we have only layer one networking we know that layer one focuses on the physical shared medium and it focuses on the standards for transmitting onto the medium and receiving from the shared medium so all devices which are part of the same layer one network need to be using the same layer one medium and device standards generally this means a certain type of network card and a certain type of cable or it means why vicar's using a certain type of antennas and frequency ranges what layer one doesn't provide is any form of access control of the shared medium and it doesn't give us uniquely identifiable devices and this means we have no method for device to device communication everything is broadcast using transmission onto the shared physical medium.

Now in the next video of this series I'm going to be stepping through layer two which is the data link layer and this is the layer which adds a lot of intelligence on top of layer one and allows device to device communication and it's layer two which is used by all of the upper layers of the OSI model to allow effective communication but it's important that you understand how layer one works because this physically is how data moves between all devices and so you need to have a good fundamental understanding of layer one.

Now this seems like a great place to take a break so I'm going to end this video here so go ahead and complete this video and then when you're ready I look forward to you joining me in the next part of this lesson series where we'll be looking at layer two or the data link layer.

Welcome back, this is part two of this lesson.

We're going to continue immediately from the end of part one.

So let's get started.

Now the only thing that remains is just to test out this configuration.

And to do that we're going to launch an EC2 instance into the WebA subnet.

So click on services and just type EC2 to move across to the EC2 console.

Now once we're on the EC2 console, just click on launch instance.

Then you'll be taken to the launch instance console.

Into the name box, just go ahead and type a4l-bastian.

Scroll down and we're going to create a bastion instance using Amazon Linux.

So click on Amazon Linux.

In the dropdown below, go ahead and select the latest version of Amazon Linux.

Just make sure that it does say free tier eligible on the right of this dropdown.

Assuming that's all good, just below that make sure that in the architecture dropdown it's set to 64-bit x86.

Moving down further still, under instance type, just make sure that this is set to a free tier eligible instance.

It should default to T2.micro or T3.micro.

Depending on your region, either of these could be free tier eligible.

In my case it's T2.micro, but whatever your shows, just make sure that it's similar sized and says free tier eligible.

Now directly below that, under key pair, just click in this box.

You should at this point in the course have a key pair creator called a4l.

If you do, go ahead and select that key pair in the box.

If you don't, don't worry, you can just go ahead and click on create new key pair.

Enter a4l into the key pair name, select RSA, and then select PEM for the private key format and click on create key pair.

This will download the key pair to your local machine and then you can continue following along with this video.

So select that from the dropdown.

Directly below, under network settings click on edit.

This instance is going to go into the animals for live VPC.

So click on the VPC dropdown and select a4l-vpc1.

Directly below that, click in the subnet dropdown and we want to go ahead and look for sn-web-a.

So select the weba subnet.

This should change both of the dropdowns below.

So auto assign public IP and auto assign IPv6 IP to enable.

So just make sure that both of those are set to enable.

Directly below this, make sure that create security group is checked.

We're going to create a new security group.

Under security group name, just go ahead and enter a4l-bassian-sg and then put that same text in the description box directly below.

Now all of these defaults should be good.

Just make sure it's set to SSH, source anywhere.

Make sure that 0.0.0.0/0 and double colon 4/0 are both present directly below source.

Everything else looks good.

We can accept the rest of the defaults.

Just go ahead and click on launch instance.

Then click on instances at the top left of the screen.

At this point the instance is launching and we'll see that a4l-bassian is currently running.

We'll see the status check is showing initializing.

So we need to give this instance a few minutes to fully provision.

So go ahead and pause this video and we're going to resume it once this instance is ready to go.

And it has two out of two status checks.

So our instance is now showing two out of two status checks.

And that means everything's good and we're ready to connect.

Now if you select the instance, you'll see in the details pane below how it has a public IP version 4 address, a private IP version 4 address, public and private IP version 4 DNS.

And if we scroll down, you'll see lots of other information about this instance.

Now we're only concerned with the public IP version 4 address.

We're going to go ahead and connect to this instance this time using a local SSH client on our machine.

So right click and then select connect.

Now if we want to quickly connect into this instance, we can choose to use EC2 instance connect, which is a way to connect into the instance using a web console.

Now this does need an instance with a public IP version 4 address, but we have allocated a public address.

So if we wanted to, we can just make sure that the username is correct.

It should be EC2-user.

If we hit connect, it will open up a connection to this instance using a web console.

And this is often much easier to connect to EC2 instances if you don't have access to a local SSH client, or if you just want to quickly connect to perform some administration.

We can also connect with an SSH client.

If we select SSH client, it gives us the commands to run in order to connect to this EC2 instance.

So right at the bottom is an example connect command.

So SSH, we pick the key to use and then we pick the user at and then the public IP version 4 DNS.

So if we copy that into our clipboard and then move across to our terminal or command prompt, move into the folder where you downloaded the SSH keybearant to, in my case downloads, and paste in that command and press enter, that should connect us to the EC2 instance.

We'll have to verify the fingerprint, so we need to verify the authenticity of this host.

For this purpose, we can just go ahead and answer yes and press enter.

Now if it's the first time we're connecting using a particular key, and if you're running either macOS or Linux, you might be informed that the permissions on the key are too open.

In this case, the permissions are 0, 6, 4, 4, which are too open and we get this error.

Now it's possible to correct that if we move back to the AWS console.

It also gives us the command to correct these permissions.

So CHmod, space 400, space and then the name of the key.

So I'm going to copy that into my clipboard and move back to my terminal, paste that in and press enter, and that will correct those permissions.

Now if I get the connection command again, so copy that into my clipboard, and this time I'll paste it in and press enter and now I will be connected to this EC2 instance.

Now if you're doing this demonstration on Windows 10, you probably won't have to correct those permissions.

This is something specific to macOS or Linux.

So whenever you're connecting to EC2 instances which have a public IP version 4 address, you've always got the ability to use either EC2 instance connect or a local SSH client.

Now the third option which is session manager, this is a way that you can connect to instances even if they don't have public IP version 4 addressing.

And I'll be detailing this product fully later on in the course because there is some additional configuration that's required.

Now this bastion host, it's an EC2 instance and it does fall under the free tier.

So because it's a T2.micro or whatever type of instance you picked which falls under the free tier, you're not going to be billed for any usage of this instance in a given month.

Now as a general rule, as you're moving through the course, if you're ever intending to take a break, then you always have the option of deleting all of the infrastructure that you've created within a specific demo lesson.

So most of the more complex demo lessons that you'll have moving through the course, at the end of every demo lesson there will be a brief set of steps where I explain how to clean up the account and return it into the same state as it was at the start of the lesson.

But in certain situations I might tell you that one option is not to delete the infrastructure.

Whether you do delete it or not depends on whether you're intending to complete the next demo straight away or whether you're taking a break.

Now in this particular case I'm going to demonstrate exactly how you can clear up this infrastructure. [background noise] [background noise] In the next demo lesson you're going to be continuing using this structure, but I'm going to demonstrate how you can automate the creation using a CloudFormation template.

To clear up this infrastructure though, go ahead, right click on this bastion host and select terminate instance.

You'll need to click terminate to confirm and that will terminate and delete the instance.

You won't be charged for any further usage of that instance.

We need to wait for that instance to fully terminate, so pause the video and wait for it to move into a terminated state and then we can continue.

So that instance is terminated and now that that's done, we can click on services and move across to the VPC console and we're going to delete the entire Animals for Life VPC.

And don't worry, in the next demo lesson I'll explain how we can automate the creation.

So for now and in the course we're going to be using much more automation so that anything that you've done previously, we're going to automate the creation and focus your valuable time only on the things that you've just learned.

So click on your VPCs.

It should list two VPCs, the default one and the Animals for Life VPC.

Select the Animals for Life VPC, click on Actions and then delete the VPC.

Now this is going to delete all of the resources that are associated with this VPC.

So the internet gateway, the route tables and all of the subnets that you've created as part of the demo lessons to this point in the course.

So go ahead and type delete and then click delete to confirm that process and that will fully tidy up the account and return it into the same state as it was at the start of the VPC section.

Now with that being said, this is the end of this lesson.

You've successfully converted three subnets, so Web A, Web B and Web C to be public and you've done that by creating an internet gateway, associating that with the VPC, creating a route table, associating that with those subnets, adding two routes, pointing those routes at the internet gateway and then configuring those subnets to allocate a public IP version for address to any resources launched into those subnets.

So that's the same set of steps that you'll need to do to make any subnets public from an IP version for perspective in future.

So this is going to be the same tasks that you would use in larger production projects.

Although in production, you would probably automate it and don't just show you how to do that as you move through the course.

Now at this point, you've finished everything that you need to do in this demo lesson, so great job.

You've actually created something that is production ready and production useful.

Over the remainder of this section of the course, we're going to refine the design that we've got and add additional capabilities.

So in the upcoming lessons, I'll be talking about network address translation and how that can be used to give private EC2 instances access to the internet for things like software updates.

We'll be talking about the security of subnets using network access control lists known as knuckles and much, much more.

But you're doing a fantastic job so far.

This is not a trivial thing that you've implemented to this point.

So really great job.

But at this point, just go ahead and complete the video.

And then when you're ready, I look forward to you joining me in the next. [no audio]

Welcome back.

This demo is going to bring together some really important theory and architecture that you've learned over the past few lessons.

What we're starting this demo lesson with is this architecture.

We have our VPC, the Animals for Life VPC in US East 1.

It uses the 10.16.0.0/16 side range.

It has 12 subnets created inside it, over three AZs with four tiers, Reserve, DB, Application and Web.

Now currently all the subnets are private and can't be used for communication with the internet or the AWS public zone.

In this demo we want to reconfigure the VPC to allow that.

So the first step is to create an internet gateway and attach it.

To do that, I'm going to move across to my desktop.

Now to do this in your environment, you'll need the VPC and subnet configuration as you set it up in the previous demo lesson.

So that configuration needs to be in place already.

You need to be logged in as the I am admin user of the management account of the organization and have the Northern Virginia region selected, so US - East - 1.

So go ahead and move across to the VPC console.

Now this should already be in the recently visited services because you were using this in the previous demo lesson, but if it's not visible just click in the services drop down, type VPC and then click to move to the VPC console.

Now if you do still have the configuration as it was at the end of the previous demo lesson, you should be able to click on subnets on the menu on the left and see a list of lots of subnets.

You'll see the ones for the default VPC without a name.

And if you have the correct configuration, you should see a collection of 12 subnets, 3 application subnets, 3 database subnets, 3 reserved subnets and then 3 web subnets.

So all of these should be in place within the Animals for Life VPC in order to do the tasks within this demo lesson.

So I'm going to assume from this point onwards that you do have all of these subnets created and configured.

Now what we're going to be doing in this demo lesson is configuring the 3 web subnets, so web A, web B and web C, to be public subnets.

Being a public subnet means that you can launch resources into the subnet, have them allocated with a public IP version 4 address and have connectivity to and from the public IP version 4 internet.

And in order to enable that functionality, there are a number of steps that we need to perform and I want you to get the practical experience of implementing these within your own environment.

Now the first step to making subnets public is that we need an internet gateway attached to this VPC.

So internet gateways, as I talked about in the previous theory lesson, are highly available gateway objects which can be used to allow public routing to and from the internet.

So we need to create one.

So let's click on internet gateways on the menu on the left.

They'll already be an internet gateway in place for the default VPC.

Remember when you created the default VPC, all this networking infrastructure is created and configured on your behalf.

But because we created a custom VPC for animals for life, we need to do this manually.

So to do that, go ahead and click on create internet gateway.

We're going to call the internet gateway A4L, so animals for life, VPC 1, which is the VPC we're going to attach it to and then IGW for internet gateway.

So A4L-VPC1-IGW.

Now that's the only information that we need to enter, so scroll down and click on create internet gateway.

Internet gateways are initially not attached to a VPC and we can tell that because it's initially in.

We need to attach this to the animals for life VPC.

So click on actions and then attach to VPC inside the available VPCs box.

Just click and then select A4L-IVAN-VPC1.

Once selected, go ahead and click on attach internet gateway and that will attach our brand new internet gateway to the animals for life VPC.

And that means that it's now available within that VPC as a gateway object, which gives the VPC the capability to communicate to and receive communications from the public internet and the AWS public zone.

Now the next step is that we want to make all the subnets in the web tier public, so the services deployed into these subnets can take advantage of this functionality.

So we want the web subnets to be able to communicate to and receive communications from the public internet and AWS public services.

Now there are a number of steps that we need to do to accomplish this.

We need to create a route table for the public subnets.

We need to associate this route table with the three public subnets, so web A, web B and web C and then we need to add two routes to this route table.

One route will be a default route for IP version 4 traffic and the other will be a default route for IP version 6 traffic.

And both of these routes for their target will be pointing at the internet gateway that you've just created and attached to this VPC.

Now this will configure the VPC router to forward any data intended for anything not within our VPC to the internet gateway.

Finally, on each of these web subnets will be configuring the subnet to auto assign a public IP version 4 address and that will complete the process of making them public.

So let's perform all of these sets of configuration.

So now that we're back at the AWS console, we need to create a route table.

So go ahead and click on route tables on the menu on the left and then we're going to create a new route table.

First we'll select the VPC that this route table will belong to and it's going to be the animals for life.

If a VPC, so go ahead and select that VPC and then we're going to give this route table a name.

And I like to keep the naming scheme consistent, so we're going to use A4, L4, animals for life and then a hyphen.

VPC1 because this is the VPC the route table will belong to and then hyphen RT for route table and then hyphen and then web because this route table is going to be used for the web subnets.

So go ahead and create this route table and click route tables on the menu on the left.

If we select the route table that we've just created, so that's the one that's called A4, L hyphen VPC1 hyphen RT hyphen web and then just expand this overview area at the bottom.

We'll be able to see all the information about this route table.

Now there are a number of areas of this which are important to understand.

One is the routes area which lists all the routes on this route table and the other is subnet associations.

This determines which subnets this route table is.

So let's go to subnet associations and currently we can see that it's not actually associated with any subnets within this VPC.

We need to adjust that so go ahead and edit those associations and we're going to associate it with the three web subnets.

So you need to select web A, web B and web C.

Now notice how all those are currently associated with the main route table of the VPC.

Remember a subnet can only be associated with one route table at a time.

If you don't explicitly associate a route table with a subnet then it's associated with the main route table.

We're going to change that.

We're going to explicitly associate this new route table with the web A, web B and web C subnets.

So go ahead and say that.

So now this route table has been associated with web A, web B and web C.

Those subnets are no longer associated with the main route table of the VPC.

So now we've configured the association as a route to routes.

And we can see that this route table has two local routes.

We've got the IP version 4 side of the VPC and the app.

IP version 6 side of the VPC.

So these two routes on this route table will mean that web A, web B and web C will know how to direct traffic towards any other IP version 4 or IP version 6 addresses within this VPC.

Now these local routes can never be adjusted or removed, but what we can do is add additional routes.

So we're going to add two routes, a default route for IP version 4 and a default route for IP version 6.

So we'll do that.

We'll start with IP version 4.

So we'll edit those routes and then we'll add a route.

The format for the IP version 4 default route is 0.0.0.0/0.

And this means any IP addresses.

Now I've talked elsewhere in the course how there is a priority to routing.

Within a VPC there's a more specific route always takes priority.

So this route, the /16 is more specific than this default route.

So this default route will only affect IP version 4 traffic, which is not matched by this local route.

So essentially anything which is IP version 4, which is not destined for the VPC, will use this default route.

Now we need to pick the internet gateway as the target for this route.

So click in the target box on this row, select internet gateway.

There should only be one that's highlighted.

That's the Animals for Life internet gateway you created moments ago.

So select that and that means that any IP version 4 traffic which is not destined for the VPC side of it. [siren] Range will be sent to the Internet Gateway.

Now we're going to do the same for IPv6.

So go ahead and add another route.

And the format for IPv60 default routes is double colon, forward slash zero.

And this is the same architecture. [siren] It essentially means this matches all IPv6 addresses, but it's less specific than the IPv6 and version 6 local route on this top row.

So this will only be used for any IPv6 addresses which are not in the IPv6 VPC side range.

So go ahead and select Target, go to Internet Gateway, and select the Animals for Life Internet Gateway.

And once you've done both of those, go ahead and click on Save Changes.

Now this means that we now have two default routes, an IPv4 default route, and an IPv6 default route.

So this means that anything which is associated with these route tables will now send any unknown traffic towards the Internet Gateway.

But what we need to do before this works is we need to ensure that any resources launched into the Web A, Web B, or Web C subnets are allocated with public IPv4 addresses.

To do that, go ahead and click on Subnets.

In the list, we need to locate Web A, Web B, and Web C.

So we'll start with Web A, so select Web A, click on Actions, and then Edit Subnet Settings.

And this time, we're going to modify this subnet so that it automatically assigns a public IPv4 address.

So check this box into the Save, and that means that any resources launched into the Web A subnet will be allocated with a public IPv4 address.

Now we need to follow the same process for the other web subnets, so select the Web B subnet, click on Actions, and then Edit Subnet Settings.

Enable IPv4, click on Save, and then do that same process for Web C.

So locate Web C, click on Actions, and then Edit Subnet Settings.

And then enable public IPv4 addresses and click on Save.

So that's all the network configuration done.

We've created an Internet Gateway.

We've associated the Internet Gateway with the Animals for IPPC.

We've created a Routetable for the web subnets.

We've associated this Routetable with the web subnets.

We've added default routes onto this Routetable, pointing at the Internet Gateway as a default IPv4 and IPv6 route.

And then we've enabled the allocation of public IPv4 addresses for Web A, Web B, and Web C.

Okay, so this is the end of Part 1 of this lesson.

It was getting a little bit on the long side, and so I wanted to add a break.

It's an opportunity just to take a rest or grab a coffee.

Part 2 will be continuing immediately from the end of Part 1.

To go ahead, complete the video, I'm ready, join me in part two.

Welcome back.

And in this lesson, I want to talk about how routing works within a VPC and introduce the internet gateway, which is how we configure a VPC so that data can exit to and enter from the AWS public zone and public internet.

Now, this lesson will be theory where I'm going to introduce routing and the internet gateway to the architecture behind both those things, as well as jump boxes also known as Bastion hosts.

In the demo lesson, which immediately follows this one, you'll get the opportunity to implement an internet gateway yourself and the animals for life at VPC and fully configure the VPC with public subnet that allow you to connect to that jump box.

So let's get started.

We've got a lot to cover.

A VPC router is a highly available device which is present in every VPC, both default or custom, which moves traffic from somewhere to somewhere else.

It runs in all the availability zones that the VPC uses and never needs a way about its availability.

It simply works.

The router can be networked in every subnet, and the network is just one address of the subnet.

By default in a custom VPC, without any other configuration, the VPC router will simply route traffic between subnets in that VPC.

If an EC2 instance in one subnet wants to communicate with something in another subnet, the VPC router is the thing that moves the traffic between subnets.

Now, the VPC router is controllable.

You create route tables which influence what's to do with traffic when it leaves a subnet.

So just to be clear, the route table that's associated with a subnet defines what the VPC router will do when data leaves that subnet.

A VPC is created with what's known as a main route table.

If you don't explicitly associate a custom route table with a subnet, it uses the main route table of the VPC.

If you do associate a route table that you create with a subnet, then when you associate that, the main route table is disassociated.

A subnet can only have one route table associated with it at any one time, but a route table can be associated with many subnets.

A route table looks like this in the user interface.

In this case, this is the main route table for this specific VPC.

And a route table is just a list of routes.

This is one of those routes.

When traffic leaves the subnet that this route table is associated with, the VPC router reviews the IP packets.

And remember, I said that a packet had a source address and a destination address, as well as some data.

The VPC router looks at the destination address of all packets leaving the subnet.

And when it has that address, it looks at the route table and it identifies all the routes which match that destination address.

And it does that by checking the destination field at the route.

This destination field determines what destination the route matches.

Now, the destination field on a route could match exactly one specific IP address.

It could be an IP with a /32 prefix.

And remember, that means that it matches one single IP.

But the destination field on a route could also be a network match.

So matching an entire network of which that IP is part.

Or it could be a default route.

Remember, for IP version 4, I mentioned that 0.0.0.0.0/0 matches all IP version 4 IP addresses.

That's known as a default route, a catchall.

In the case where traffic leaving a subnet only matches one route, then that one route is selected.

If multiple routes match, so maybe there's a specific /32 IP match, maybe there's a /16 network match, and maybe there's a 0.0.0.0/0 default match, well then the prefix is used as a priority.

The higher the prefix value, the more specific the route is and the higher priority that that route has.

So the higher the prefix, all the way up to the highest priority of /32, that is used to select which route applies when traffic leaves a subnet.

Once we get to the point where a single rule in a route table is selected, either the sole route that applies or the one with the highest priority, then the VPC router forwards that traffic through to its destination, which is determined by the target field on the route.

And the target field will either point at an AWS gateway, or it will say, as with this example, local.

And local means that the destination is in the VPC itself, so the VPC router can forward the traffic directly.

All route tables have at least one route, the local route.

This matches the VPC side range, and it lets the VPC router know that traffic destined for any IP address in the VPC side range is local and it can be delivered directly.

If the VPC is also IPv6 enabled, people also have another local route matching the IPv6 side for the VPC.

As is the case with this example, that bottom route, beginning 2600, that is an IPv6 local route.

That's the IPv6 side of this specific VPC.

Now, these local routes can never be updated.

They're always present, and the local routes always take priority.

They're the exception to that previous rule about the more specific the route is, the higher the priority.

Local routes always take priority.

For the exam, remember the route tables are attached to zero or more subnets.

A subnet has to have a route table.

It's either the main route table of the VPC or a custom one that you've created.

A route table controls what happens to data as it leaves the subnet or subnets that that route table is associated with.

Local routes are always there, uneditable, and match the VPC IPv4 or VPC side range.

For anything else, higher, prefix values are more specific than they take priority.

The way the route works is it matches a destination IP, and for that route, it directs traffic towards a specific target.

Now, a default route, which I'll talk about shortly, is what happens if nothing else matches.

Now, an internet gateway is one of the most important add-on features available within a VPC.

It's a regionally resilient gateway which can be attached to a VPC.

I've highlighted the words "region" and "resilience" because it always comes up in the exam.

You do not need a gateway per availability zone.

The internet gateway's resilient by design.

One internet gateway will cover all of the availability zones in the region which the VPC is using.

Now, there's a one-to-one relationship between internet gateways and the VPC.

A VPC can have no internet gateways which makes it entirely private, or it can have one internet gateway.

Those are the two choices.

An internet gateway can be created and not attached to a VPC, so it can have zero attachments, but it can only ever be attached to one VPC at a time, at which point it's valid in all of the availability zones that the VPC uses.

Now, the internet gateway runs from the border of the VPC and the AWS public zone.

It's what allows services inside the VPC, which are allocated with public IP version 4 addresses or IP version 6 addresses, to be reached from the internet and to connect to the AWS public zone or the internet.

Of course, the AWS public zone is used if you're accessing S3, SQS, SNS, or any other AWS public services from the VPC.

Now, it's a managed gateway and so AWS handles the performance.

From your perspective as an architect, it simply works.

Now, using an internet gateway within a VPC, it's not all that complex.

It's a simplified VPC diagram.

First, we create and attach an internet gateway to a VPC.

This means that it's available for use inside the VPC.

We can use it as a target within route tables.

So then we create a custom route table and it's within route tables.

So then we create a custom route table and we associate it with a web subnet.

Then we add IP version 4 and optionally IP version 6 default routes to the route table with the target being the internet gateway.

Then finally, we configure the subnet to allocate IP version 4 addresses and optionally IP version 6 by default.

And at that point, once we've done all of those actions together, the subnet is classified as being a public subnet and any services inside that subnet with public IP addresses can communicate to the internet and vice versa and they can communicate with the AWS public zone as long as there's no other security limitations that are in play.

Now, don't worry if this seems complex.

You'll get to experience it shortly in the upcoming demo lesson.

But before that, I want to talk about how IP version 4 addressing actually works inside the VPC because I've seen quite a few difficult questions on the exam based around IP version 4 addressing and I want to clarify exactly how it works.

So conceptually, this is how an EC2 instance might look if it's using IP version 4 to communicate with a software update server of some kind.

So we've got the instance on the left with an internet gateway in between and let's say it's a Linux EC2 instance trying to do some software updates to a Linux update server that's located somewhere in the public internet zone.

So the instance has a private IP address of let's say 10.16.16.20 and it also has an IP version 4 public address that's assigned to it of 43.250.192.20.

Only that's not how it really works.

This is another one of those little details which I try to include in my training courses because it really comes invaluable for the exam.

What actually happens with public IP version 4 addresses is that they never touch the actual services inside the VPC.

Instead, when you allocate a public IP version 4 address, for example, to this EC2 instance, a record is created which the internet gateway maintains.

It links the instance's private IP to its allocated public IP.

So the instance itself is not configured with that public IP.

That's why when you make an EC2 instance and allocate it a public IP version 4 address, inside the operating system, it only sees the private IP address.

Keep this in mind for the exam that there are questions which will try to trip you up on this one.

For IP version 4, it is not configured in the OS with the public IP address.

So let's look at the flow of data.

How does this work?

Well, when the Linux instance wants to communicate with the Linux software update server, it creates a packet of data.

Now obviously it probably creates a lot of packets.

Well, let's focus on one for now because it keeps the diagram nice and simple.

The packet has a source address of the EC2 instance and a destination address of the Linux software update server.

So at this point, the packet is not configured with any public addressing.

This packet would not be routable across the public internet.

It could not reach the Linux update server.

That's really important to realize.

Now the packet leaves the instance and because we've configured a default route, it arrives at the internet gateway.

The internet gateway sees that this packet is from the EC2 instance because it analyzes the source IP address of that packet.

And it knows that this instance has an associated public IP version 4 address.

And so it adjusts the packet.

It changes the packet's source IP address to the public IP address that's allocated to that instance.

And this IP address, because it's public, is routable across the internet.

So the internet gateway then forwards the updated packet onto its destination.

So as far as the Linux software update server is concerned, it's receiving a packet from a source IP address of 43.250.192.20.

It knows nothing of the private IP address of the EC2 instance.

Now on the way back, the inverse happens.

The Linux software update server wants to send a packet back to our EC2 instance.

But as far as it's concerned, it doesn't know about the private address.

It just knows about the public address.

So the software update server sends a packet back, addressed to the instance's public IP address with its source address.

So it thinks that the real IP address of the instance is this 43.250.192.20 address.

Now this IP address actually belongs to the internet gateway.

And so it travels over the public internet and it arrives at the internet gateway, which then modifies this packet.

It changes it.

It changes the destination address of this packet from the 43 address to the original IP address of the EC2 instance.

And it does this because it's got a record of the relationship between the private IP and the allocated public IP.

So it just changes the destination to the private IP address of the instance.

And then it forwards this packet through the VPC network to the original EC2 instance.

So the reason I wanted to highlight this is because at no point is the operating system on the EC2 instance aware of its public IP.

It just has a private IP.

Don't fall for any exam questions which try to convince you to assign the public IP version 4 address of an EC2 instance directly to the operating system.

It has no knowledge of this public address.

Configuring an EC2 instance appropriately using IP version 4 means putting the private IP address only.

The public address never touches the instance.

For IP version 6, all addresses that AWS users are natively, publicly, routable.

And so in the case of IP version 6, the operating system does have the IP address version 6 address configured upon it.

That's the publicly routable address.

And all the internet gateway does is pass traffic from an instance to an internet server.

And then back again, it doesn't do any translation.

Now, before we implement this in the demo lesson, I just want to briefly touch upon bastion hosts and jump boxes.

At a high level, bastion hosts and jump boxes are one and the same.

Essentially, it's just an instance in a public subnet inside of VPC.

And architecturally, they're used to allow incoming management connections.

So all incoming management connections arrive at bastion hosts or jump box.

And then once connected, you can then go on to access internal only VPC resources.

So bastion hosts and jump boxes are generally used either as a management point or as an entry point for private only VPCs.

So if your VPC is a highly secure private VPC, you'll generally have a bastion host or a jump box being the only way to get access to that VPC.

So it's essentially just an inbound management point.

And you can configure these bastion hosts or jump boxes to only accept connections from certain IP addresses, to authenticate with SSH, or to integrate with your on-premises identity servers.

You can configure them exactly how you need, but at a real core architectural level, they are generally the only entry point to a highly secure VPC.

And historically, they were the only way to manage private VPC instances.

Now, there are alternative ways to do that now, but you will still find bastion hosts and jump boxes do feature on the exam.

OK, so that's all the theory that I wanted to cover in this lesson.

It's now time for a demo.

In the next lesson, we're going to implement the Internet gateway in the Animals for Life VPC.

We'll create it, we'll attach it to the VPC, we'll create a custom route table for the web subnets, we'll create two routes in that route table, one for IP version 4 and one for IP version 6.

And both of these will point at the Internet gateway as a target.

We'll associate that route table with the web tier subnets, configure those subnets to allocate public IP version 4 addresses, and then launch a bastion host into one of those web subnets.

And if all goes well, we will be able to connect to that instance using our SSH application.

So I think this demo lesson is going to be really interesting and really exciting.

It's the first time that we're going to be stepping through something together that we could qualify as production like.

Something that you could implement and would implement in a production ready VPC.

So go ahead, complete this video, and when you're ready, join me in the demo lesson.

Welcome back and in this demo lesson you're going to create all of the subnets within the custom VPC for animals for life.

So we're going to create the subnets as shown on screen now.

We're going to be creating four subnets in each availability zone.

So that's the web subnet, the application subnet, the database subnet and the reserved subnet.

And we're going to create each of those four in availability zone A, B and C.

Now before we get started, attached to this lesson is a link to this document.

Now this is a list of all the details of the subnet you're going to create in this demo lesson.

So we've got a reserved subnet, a database subnet, an app subnet and a web subnet.

And we've got each of those in availability zone A, B and C.

Now in terms of what this document contains, we have a subnet name, then we have the IP range that subnet will use, the availability zone that subnet is within and then this last value and we'll talk about what this is very shortly.

This relates to IP version 6.

Now you'll notice that for each subnet this is a unique value.

You're going to need this to configure the IP version 6 configuration for each of the subnets.

Now let's get started.

So let's move to the AWS console and you need to be within the VPC console.

So if you're not already there, go ahead and type that in the search box at the top and then click to move to the VPC console and then click on subnets.

And once you've done that, go ahead and click on create subnet.

Now the newest version of the user interface allows you to create multiple subnets at a time and so we're going to create all four of the subnets in each availability zone.

So we'll do this three times, one for availability zone A, one for B and one for C.

So we'll get started with availability zone A and first we need to select the VPC to use.

So click in the VPC ID drop down and select the animals for live VPC.

Once you've done that, we're going to start with subnet one of one.

So let's move to the subnet's document.

So it's these subnets that we're going to create and we'll start with the reserved subnet.

So copy the name of the subnet into your clipboard and paste it into subnet name.

Then change the availability zone to AZA and then make sure IPv4 is set to manual and move back to the subnet's document and copy the IP range that we're going to use and paste that into this box.

Then scroll down again and make sure manual input is selected for IPv6.

Then click the drop down and select the IPv6 range for the VPC.

Now the VPC uses a /56 IPv6 range.

Because we need our subnet to fit inside this, we're going to make the individual subnet ranges much smaller.

So what I'll need you to do and you'll need to do this each time is to click on the down arrow and you'll need to click on the down arrow twice.

The first time will change it to a /60 and the second time to a /64.

Now note in my case how I have 9, 6, 0, 0 and if I click on this right arrow, it increments this value by one each time.

Now this value corresponds to the value in the subnet's document, so in this case 0, 0.

By changing this value each time, it means that you're giving a unique IPv6 range to each subnet.

So in this case, start off by leaving this set to 0, 0.

And once you've done that, you can click on add new subnet.

And we're going to create the next subnet.

So in this case, it's sn-db-a, so enter that name, change the availability zone to A, manual input for IPv4, copy the IP range for DBA into your clipboard, paste that in, manual for IPv6, change the VPC range in the drop down, click the down arrow twice to set this to /64, and then change this value to 0, 1.

And again, this matches the IPv6 value in the subnet's document.

Then we'll do the same process for the third subnet, so we're going to add a new subnet.

This time the name is sn-app-a, enter that, availability zone A, manual for IPv4, and then paste in the IP range, manual for IPv6, and select the VPC range, and then change the subnet block to /64 by clicking the down arrow twice.

And then click the right arrow twice to set 0, 2 as the unique value for the subnet.

And again, this matches the value in the subnet's document.

Then lastly, we're going to do the same thing for the last subnet in availability zone A, so we're going to add a new subnet.

This time it's web A, so copy and paste that into the subnet name box, set availability zone A, manual for IPv4, copy and paste the range from the subnet's document, manual for IPv6, select the IPv6 range from the VPC, click the down arrow twice to set the appropriate size for the subnet IPv6 range, and then click on the right arrow to change this value to 0, 3.

Now that's all the subnet's created, all four of them in availability zone A, so we can scroll all the way down to the bottom, and click on create subnet, and that's going to create all four of those subnets.

We can see those in this list.

Now we're going to follow that same process for availability zone B, so click on create subnet, change the VPC to the animals for IPvc, and now we're going to start moving through quicker.

So first we're going to do the reserve B subnet, so copy the name, paste that in, set the availability zone this time to B, manual for IPv4, paste in the range, manual for IPv6, select the VPC range in the drop-down, click on the down arrow twice to set the /64, and then click on the right arrow and set to 0, 4 as the unique value.

Scroll down, click add new subnet, next is DBB, enter that, availability zone B, manual for IPv4, enter the appropriate range, and just take note of the IPv6 value because then we don't have to keep switching backwards and forwards to this document in this case it's 0, 5, paste in the IPv4 range, manual for IPv6, select the VPC range in the drop-down, down arrow twice, and then the left arrow and set to 0, 5, which is the unique value for the subnet, click add new subnet, this time it's app B, enter that name, availability zone B, manual for IPv4, you'll need to enter the IP range for app B, and again pay attention to the fact that the unique value for the subnet is 0, 6, manual for IPv6, select the VPC range in the drop-down, down arrow twice, right arrow until it says 0, 6, and then add new subnet again, we're going to do the last one, this time it's WebB, enter that name, availability zone B, and drop-down, manual for IPv4, copy and paste the IP range, and pay attention to 0, 7, which is the IPv6 unique value, enter the IPv4 subnet range in this box, manual for IPv6, select the VPC range in the drop-down, down arrow twice, and then right arrow until it says 0, 7, and now we've got all four subnets in AZB, so click on create subnet, and then we're going to do this one last time for availability zone C, so click on create subnet, select the animals for IPPC in the drop-down, and we're going to follow the same process, so for subnet 1 it will be SN_reserved-C, availability zone C, you'll need to enter the IPv4 range, pay attention to the IPv6 unique value, which is 0, 8, paste in that range in the box, manual for IPv6, select the VPC range, down arrow twice, set it to /64, and then right arrow until it says 0, 8, scroll down, add a new subnet, next is DVC, so enter that, availability zone C, and do the same thing as before, we'll need the IP range, and the IPv6 unique value, so 0, 9, enter that, manual for IPv6, select the VPC range in the drop-down, down arrow twice, and then click the right arrow until 0, 9 is selected, add a new subnet, then the application subnet, copy that, paste it in, availability zone C, get the IPv4 range and the unique value of IPv6, now note this is hexadecimal, so 0, 8 directly follows 0, 9, so pay attention to that, go back, paste in the IPv4 range, manual for IPv6, select the VPC range, down arrow twice to select /64, and then right arrow until you get 0, 8, then one last time, click on add new subnet, go back to the subnet document, Web C, availability zone C, get the IPv4 range and note the unique IPv6 value, paste that in, select the IPv6 range for the VPC, down arrow twice to select /64, and then right arrow all the way through to 0, B, at that point you can go ahead and click on create subnet, and that's created all four subnets in availability zone C, and all of the subnets now that are within the annuals for IPv6, at least those in AZA, AZB, and AZC, once again we're not going to create the ones in AZD which are reserved for future growth, now there's one final thing that we need to do to all of these subnets, so each of these subnets is allocated with an IPv6 range, however it's not set to auto-allocate IPv6 addressing to anything created within each of these subnets, now to do that go ahead and select SN-AP-A, click on actions, and then edit subnet settings, and I want you to check the box to say enable auto-assign IPv6 addresses, once you've done that scroll to the bottom and click on save, so that's one subnet that you've done that for, next I want you to do it for app B, follow the same process, actions, edit subnet settings, auto-assign IPv6, click on save, notice how we're not touching the IPv4 setting, we'll be changing that as appropriate later, select SN-AP-AP-A and see and again edit subnet settings, enable auto-assign IPv6, and click on save, then we're going to do the same for the database subnets, so dba, edit subnet settings, enable IPv6 and save, then dbb, check this box, save, then dbc, edit subnet settings, check this box, save, now we'll do the reserve, so reserve A, then reserve B, and then reserve C, and then finally we're going to do the web subnets, so we'll start with A, again make sure you're only changing the IPv6 box, say that, do the same with web B, and then once that's done we'll scroll down and do the final subnet, so web C, same process, IPv6, and save, and at this point you've gone through the very manual process of creating 12 subnets across three availability zones using the architecture that's shown on screen now, now in production usage in the real world you would automate this process, you wouldn't do this manually each and every time, but I think it's important that you understand how to do this process manually, so you can understand exactly what to select when configuring automation to achieve the same end goal, so whenever I'm using automation I always like to understand how it works manually, so that I can fully understand what it is that automation is doing, now at this point that is everything that I wanted to cover in this demo lesson, we're going to be continually evolving this design as we move through this section of the course, but at this point that is everything I wanted to do, so go ahead and complete this video and when you're ready, I'll look forward to you joining me in the next.

Welcome back.

And in this lesson, I want to continue the theme of VPC networking in AWS by covering VPC subnets.

Now subnets are what services run from inside VPCs and they're how you add structure, functionality and resilience to VPCs.

So they're an important thing to get right, both for production deployment and to do well in the exam.

So let's not waste time.

Let's jump in and get started.

In this lesson, we'll be starting off with this architecture.

This is exactly how we left off at the end of the previous lesson, a framework VPC, a skeleton.

What we'll be doing is creating an internal structure using subnets.

We'll be turning this into this.

Now if you compare this diagram to the one that I've linked previously, you might notice that the web tier subnets on the right are blue on this diagram instead of green on the diagram that I previously linked.

Now with AWS diagrams, blue means private subnets and green means public subnets.

Subnets inside a VPC start off entirely private and they take some configuration to make them public.

So at this point, the subnets which will be created on the right, so the web tier, they'll be created as private subnets.

And in the following lessons, we'll change that together.

So for now, this diagram showing them as private subnets is correct.

So what exactly is a subnet?

It's an easy, resilient feature of a VPC, a subnetwork of the VPC, a part of the VPC that's inside a specific availability zone.

It's created within one availability zone and it can never be changed because it runs inside of an availability zone.

If that availability zone fails, then the subnet itself fails and so do any services that are only hosted in that one subnet.

And as AWS Solutions Architects, when we design highly available architectures, we're trying to put different components of our system into different availability zones to make sure that if one fails, our entire system doesn't fail.

And the way that we do that is to put these components of our infrastructure into different subnets, each of which are located in a specific availability zone.

The relationship between subnets and availability zones is that one subnet is created in a specific availability zone in that region.

It can never be changed and a subnet can never be in multiple availability zones.

That's the important one to remember for the exam.

One subnet is in one availability zone.

A subnet can never be in more than one availability zone.

Logically, though, one availability zone can have zero or lots of subnets.

So one subnet is in one availability zone, but one availability zone can have many subnets.

Now, the subnet by default uses IP version 4 networking and it's allocated an IP version 4 sider.

And this sider is a subset of the VPC side block.

It has to be within the range that's allocated to the VPC.

What's more, the side of the subnet users cannot overlap with any other subnets in that VPC.

They have to be non-overlapping.

That's another topic which tends to come up all the time in the exam.

Now, a subnet can optionally be allocated an IP version 6 side block as long as the VPC also is enabled for IP version 6.

The range that's allocated to individual subnets is a /64 range.

And that /64 range is a subset of that /56 VPC.

So /56 IP version 6 range has enough space for 256 /64 ranges that each subnet can use.

Now, subnets inside the VPC can, by default, communicate with other subnets in that same VPC.

The isolation of the VPC is at the perimeter of the VPC.

Internally, there is free communication between subnets by default.

Now, we spoke in previous lessons about sizing.

So sizes of networks are based on the prefix.

For example, a /24 network allows values from 0 to 255 in the fourth octet.

Well, that's a possible 256 possible IPs.

But inside a subnet, you don't get to use them all.

Some IPs inside every VPC subnet are reserved.

So let's look at those next.

There are five IP addresses within every VPC subnet that you can't use.

So whatever the size of the subnet, the usable IPs are five less than you would expect.

Let's assume, for example, that the subnet we're talking about is 10.16.16.0/20.

So this has a range of 10.16.16.0 to 10.16.31.255.

The first address which is unusable is the network address.

The first address of any subnet is the one that represents the network, the starting address of the network, and it can't be used.

This isn't specific to the US.

It's a case for any other ID networks as well.

Nothing uses the first address on a network.

Next is what's known as the network plus one address.

The first IP after the network address.

And in AWS, this is used by the VPC router, the logical network device which moves data between subnets, and in and out of the VPC if it's configured to allow that.

The VPC router has a network interface in every subnet, and it uses this network plus one address.

Next is another AWS specific IP address which can't be used, called the network plus two address.

In a VPC, the second usable address of the VPC range is used for DNS.

In AWS, reserve the network plus two address in every subnet.

So I've put DNS and an asterisk here because I refer to this reservation as the DNS reservation.

But strictly speaking, it's the second address in a VPC which is used for DNS.

So that's the VPC range plus two.

But AWS do reserve the network plus two address in every single subnet.

So you need to be aware of that.

And there's one more AWS specific address that you can't use, and you guessed it, it's a network plus three address.

This doesn't have a use yet.

It's reserved for future requirements.

And this is the network plus three, and this is ample ten, sixteen, sixteen, sixteen, three.

And then lastly, the final IP address that can't be used in every VPC subnet is the network broadcast address.

Broadcast are not supported inside a VPC, but the last IP address in every subnet is reserved regardless.

So you cannot use this last address.

So this makes a total of five IP addresses in every subnet that you can't use, three AWS specific ones, and then the network and broadcast addresses.

So if the subnet should have 16 IDs, it actually has 11 usable IDs.

So keep this in mind, especially when you're creating smaller VPCs and subnets, because this can quickly eat up IP addresses, especially if you use small VPCs with lots of subnets.

Now a VPC has a configuration object applied to it called a DHCP option set.

DHCP stands for Dynamic Code Configuration Protocol.

It's how computing devices receive IP addresses automatically.

Now there's one DHCP option set applied to a VPC at one time, and this configuration flows through to subnets.

It controls things like DNS servers, NTP servers, NetBioh servers, and a few other things.

If you've ever managed a DHCP server, this will be familiar.

So for every VPC, there's a DHCP option set that's linked to it, and that can be changed.

You can create option sets, but you cannot edit them.

So keep in mind, if you want to change the settings, you need to create a new one, and then change the VPC allocation to this new one.

On every subnet, you can also define two important IP allocation options.

The first option controls if resources in a subnet are allocated a public IP version 4 address, in addition to their private subnet address automatically.

Now I'm going to be covering this in a lesson on routing and internet gateway, because there's some additional theory that you need to understand about public IP version 4 addresses.

But this is one of the steps that you need to do to make a subnet public.

So it's on a per subnet basis that you can set auto assign public IP version 4 addresses.

Another related option defined at a subnet level is whether resources deployed into that subnet are also given an IP version 6 address, and logically for that to work, the subnet has to have an allocation as does the VPC.

But both of these options are defined at a subnet level and flow onto any resources inside that subnet.

Okay, so now it's time for a demo.

That's all the theory that I wanted to cover in this VPC subnet lesson.

So in the demo lesson, we're going to implement the structure inside VPC together.

We're essentially going to change this skeleton VPC into a multi-tier VPC that's configured with all of these subnets.

Now it's going to be a fairly detailed demo lesson.

You can have to create all of these 12 subnets manually one by one.

And out of all the lessons, the detail really matters on this one.

We need to make sure that you configure this exactly as required so you don't have any issues in future.

Now if you do make any mistakes, I'm going to make sure that I supply a CloudFormation template with the next lesson that allows you to configure this in future automatically.

But the first time that you do this lesson, I do want you to do it manually because you need to get used to the process of creating subnets.

So controlling what the IP ranges are, being able to select which availability zone they go in, and knowing how to assign IP version 6 ranges to those subnets.

So it is worthwhile investing the time to create each of these 12 subnets manually.

And that's what we're going to do in the next demo lesson.

But at this point, go ahead and complete this lesson, and then when you've got the time, I'll see you in the next demo lesson where we'll complete the configuration of this VPC by creating the subnets.

Welcome back.

Over the remaining lessons in this section, you're going to learn how to build a complex, multi-tier, custom VPC step by step.

One of the benefits of the VPC product is that you can start off simple and layer components in piece by piece.

This lesson will focus on just the VPC shell, but by the end of this section, you'll be 100% comfortable building a pretty complex private network inside AWS.

So let's get started.

Now, don't get scared off by this diagram, but this is what we're going to implement together in this section, of course.

Right now, it might look complicated, but it's like building a Lego project.

We'll start off simple and add more and more complexity as we go through the section.

This is a multi-tier, custom VPC.

If you look at the IP plan document that I linked in the last lesson, it's using the IP address at the first range of the US Region 1 for the general account, so 10.16.0.0/16.

So the VPC will be configured to use that range.

Inside the VPC, there'll be space for four tiers running in four availability zones for a total of 16 possible subnets.

Now, we'll be creating all four tiers, so reserved, database, app, and web, but only three availability zones, A, B, and C.

We won't be creating any subnets in the capacity reserved for the future availability zone, so that's the part at the bottom here.

In addition to the VPC that we'll create in this lesson, the subnets that we'll create in the following lessons will also, as we look through the section of the course, be creating an internet gateway which will give resources in the VPC public access.

We'll be creating NAT gateways which will give private instances outgoing only access, and we'll be creating a bastion host which is one way that we can connect into the VPC.

Now, using bastion hosts is frowned upon and isn't best security practice for getting access to AWS VPCs, but it's important that you understand how not to do something in order to appreciate good architectural design.

So I'm going to step you through how to implement a bastion host in this part of the course, and as we move through later sections of the course, you'll learn more secure alternatives.

Finally, later on in the section, we'll also be looking at network access control lists on knuckles which can be used to secure the VPC, as well as data transfer costs for any data that moves in and around the VPC.

Now, this might look intimidating, but don't worry, I'll be explaining everything every step of the way.

To start with though, we're going to keep it simple and just create the VPC.

Before we do create a VPC, I want to cover some essential architectural theory, so let's get started with that.

VPCs are a regionally isolated and regionally resilient service.

A VPC is created in a region and it operates from all of the AZs in that region.

It allows you to create isolated networks inside AWS, so even in a single region in an account, you can have multiple isolated networks.

Nothing is allowed in or out of a VPC without a piece of explicit configuration.

It's a network boundary and it provides an isolated glass radius.

What I mean by this is if you have a problem inside a VPC, so if one resource or a set of resources are exploited, the impact is limited to that VPC or anything that you have connected to it.

I talked earlier in the course about the default VPC being set up by AWS using the same static structure of one subnet per availability zone using the same IP address ranges and requiring no configuration from the account administrator.

Well, custom VPCs are pretty much the opposite of that.

They let you create networks with almost any configuration, which can range from a simple VPC to a complex multi-tier one such as the one that we're creating in this section.

Custom VPCs also support hybrid networking, which let you connect your VPC to the cloud platforms as well as on-premises networks, and we'll cover that later on in the course.

When you create a VPC, you have the option of picking default or dedicated dependency.

This controls whether the resources created inside the VPC are provisioned on shared hardware or dedicated hardware.

So be really careful with this option.

If you pick default, then you can choose on a per-resource basis later on when you provision resources as whether it goes on shared hardware or dedicated hardware.

If you pick dedicated tenancy at a VPC level, then that's locked in.

Any resources that you create inside that VPC have to be on dedicated hardware.

So you need to be really careful with this option because dedicated tenancy comes at a cost premium, and my rule on this is unless you really know that you require dedicated, then pick default, which is the default option.

Now, VPC can use IP version for private and public IPs.

The private side block is the main method of IP communication for the VPC.

So by default, everything uses these private addresses.

Public IPs are used when you want to make resources public, when you want them to communicate with the public internet or the AWS public zone, or you want to allow communication to them from the public internet.

Now, VPC is allocated one mandatory private IP version for side block.

This is configured when you create the VPC, which you'll see in a moment when we actually create a VPC.

Now, this primary block has two main restrictions.

It can be at its smallest, a /28 prefix, meaning the entire VPC has 16 IP addresses, and some of those can't be used.

More on that in the next lesson when I talk about subnet, though.

At the largest, a VPC can use a /16 prefix, which is 65,536 IDs.

Now, you can add secondary IP version for side blocks after creation, but by default, at the time of creating this lesson, there's a maximum of five of those, but they can be increased by using a support ticket.

But generally, when you're thinking conceptually about a VPC, just imagine that it's got a pool of private IP version 4 addresses, and optionally, it can use public addresses.

Now, another optional configuration is that a VPC can be configured to use IP version 6 by assigning a /56 IP V6 sider to the VPC.

Now, this is a feature set which is still being enjoyed, so not everything works with the same level of features as it does for IP version 4, but with the increasing worldwide usage of IP version 6, in most circumstances, you should start looking at applying an IP version 6 range as a default.

An important thing about IP version 6 is that the range is either allocated by AWS, as in you have no choice on which range to use, or you can select to use your own IP version 6 addresses, addresses which you own.

You can't pick a block like you can with IP version 4, either let AWS assign it or you use addresses that you own.

Now, IP version 6 IPs don't have the concept of private and public, the range of IP version 6 addresses that AWS uses are all publicly routable by default.

But if you do use them, you still have to explicitly allow connectivity to and from the public internet.

So don't worry about security concerns, it just removes an admin overhead because you don't need to worry about this distinction between public and private.

Now, AWS VPCs also have fully featured DNS.

It's provided by round 53, and inside the VPC, it's available on the base IP address of the VPC plus 2.

So the VPC is 10.0.0.0, and the DNS IP will be 10.0.0.2.

Now, there are two options which are critical for how DNS functions in a VPC, so I've highlighted both of them.

The first is a setting called enable DNS host names, and this indicates whether instances with public IP addresses in a VPC are given public DNS host names.

So if this is set to true, then instances do get public DNS host names.

If it's not set to true, they don't.

The second option is enable DNS support, and this indicates whether DNS is enabled or disabled in the VPC, so DNS resolution.

If it is enabled, then instances in the VPC can use the DNS IP address, so the VPC plus 2 IP address.

If this is set to false, then this is not available.

Now, why I mention both of these is if you do have any questions in the exam or any real world situations where you're having DNS issues, these two should be the first settings that you check, switched on or off as appropriate.

And in the demo part of this lesson, I'll show you where to access those.

Speaking of which, it's now time for the demo component of this lesson, and we're going to implement the framework of VPC for the Animals for Life organization together inside our AWS account.

So let's go ahead and finish the theory part of this lesson right now, and then in the next lesson, the demo part will implement this VPC together.

So, I'm going to start with the demo part. .

Welcome back, this is part two of this lesson.

We're going to continue immediately from the end of part one, so let's get started.

That's a good starting point for our plan.

Before I elaborate more on that plan though, let's think about VPC sizing and structure.

AWS provides some useful pointers on VPC sizing, which I'll link to in the lesson text, but I also want to talk about it briefly in this lesson.

They define micro as a /24 VPC with eight subnets inside it, each subnet as a /27, which means 27 IP addresses per subnet, and a total of 216.

This goes all the way through to extra large, which is a /16 VPC with 16 subnets inside, each of which is a /20, offering 4,091 IP addresses per subnet, for a total of just over 65,000.

And deciding which to use, there are two important questions.

First, how many subnets will you need in each VPC?

And second, how many IP addresses will you need in total, and how many IP addresses in each subnet?

Now deciding how many subnets to use, there's actually a method that I use all the time, which makes it easier.

Let's look at that next.

So this is the shell of a VPC, but you can't just use a VPC to launch services into.

That's not how it works in AWS.

Services use subnets, which where IP addresses are allocated from, VPC services run from within subnets, not directly from the VPC.

And if you remember, all the way back at the start of the course where I introduced VPCs and subnets, I mentioned that a subnet is located in one availability zone.

So the first decision point that you need to think about is how many availability zones your VPC will use.

This decision impacts high availability and resilience, and it depends somewhat on the region that the VPC is in, since some regions are limited in how many availability zones they have.

So we'll have three, so we'll have more.

So step one is to pick how many availability zones your VPC will use.

Now I'll spoil this and make it easy.

I always start with three as my default.

Why?

Because it will work in almost any region.

And I also always add a spare, because we all know at some point things grow, so I aim for at least one spare.

And this means there's a minimum for availability zones, A, B, C, and the spare.

If you think about it, that means that we have to at least split the VPC into at least four smaller networks.

So if we started with a /16, we would now have four /18s.

As well as the availability zones inside of VPC, we also have tiers, and tiers are the different types of infrastructure that are running inside that VPC.

We might have a web tier, an application tier, a database tier, that makes three, and you should always add buffer.

So my default is to start with four tiers, web, application, database, and a spare.

Now the tiers you think your architecture might be different, but my default for most designs is to issue three plus a spare, to web, application, database, and then a spare for future use.

If you only used one availability zone, then each tier would need its own subnet, meaning four subnets in total.

But we also have four AZs, and since we want to take full advantage of the resiliency provided by these AZs, we need the same base networking duplicated in each availability zone.

So each tier has its own subnet in each availability zone, for web subnets, for app subnets, for database subnets, and for spares, for a total of 16 subnets.

So if we chose a /16 for the VPC, that would mean that each of the 16 subnets would need to fit into that /16.

So a /16 VPC split into 16 subnets results in 16 smaller network ranges, each of which is a /20.

Remember, each time the prefix is increased, from 16 to 17, it creates two networks, from 16 to 18, it creates four, from 16 to 19, it creates eight, from 16 to 20, it creates 16 smaller networks.

Now that we know that we need 16 subnets, we could start with a /17 VPC, and then each subnet would be a /21, or we could start with a /18 VPC, and then each subnet would be a /22, and so on.

Now that you know the number of subnets, and because of that, the size of the subnets in relation to the VPC prefix size, picking the size of the VPC, is all about how much capacity you need.

Whenever prefix you pick for the VPC, the subnets will be four steps away.

So let's move on to the last part of this lesson, where we're going to be deciding exactly what to use.

Now, Animals for Life is a global organization already, but with what's happening environmentally around the world, the business could grow significantly, and so when designing the IP plans for the business, we need to assume a huge level of growth.

We've talked about a preference for the 10 range, but avoiding the common networks and avoiding Google, would give us a 10.16 to 10.127 to use as /16 networks.

We have five regions that we're going to be assuming the business will use, three to be chosen in the US, one in Europe, and one in Australia.

So if we start at 10.16 and break this down into segments, we could choose to use 10.16 to 10.31 as US Region 1, 10.32 to 10.47 as US Region 2, 10.48 to 10.63 as US Region 3, 10.64 to 10.79 as Europe, and 10.18 to 10.95 as Australia.

That is a total of 16 /16 network ranges for each region.

Now, we have a total of three accounts right now, general, prod, and dev, and let's add one more buffer, so that's four total accounts.

So if we break down those ranges that we've got for each region, break them down into four, one for each account, then each account in each region gets four /16 ranges, you know, for four VPCs per region per account.

So I've created this PDF and I've included this attach to this lesson and in this lessons folder on the course GitHub repository.

So if you go into VPC-basics, in there is a folder called VPC-Sizing and Structure, and then in this folder is a document called A4L for Animals for Life, underscore idplan.pdf, and this is that document.

So I've just tried to document here exactly what we've done with these different ranges.

So starting at the top here, we've blocked off all these networks, these are common ranges to avoid, and we're starting at 10.16 for Animals for Life, and then starting at 10.16, I've blocked off 16 /16 networks for each region.

So US Region 1, Region 2, Region 3, Europe, and Australia, and then we're left with some of the renewed and they're reserved.

After that, of course, from 10.1 to 8 onwards, that's reserved for the Google Cloud usage, which we're uncertain about.

So all the way to the end, that's blocked off.

And then within each region, we've got three A.L.

US accounts that we know about, general, prod, and dev, and then one set for reserved future use.

So in the region, each of those accounts has four Class B networks, enough for four non-overlapping VPCs.

So feel free to look through this document, I've included the PDF and the original A.L. numbers document, so feel free to use this, adjust this for your network, and just experiment with some IP planning.

But this is the type of document that I'll be using as a starting point for any large A.L.

US deployments.

I'm going to be using this throughout this course to plan the IP address ranges whenever we're creating a VPC.

We obviously won't be using all of them, but we will be using this as a foundation.

Now based on that plan, that means we have a /16 range to use for each VPC in each account in each region, and these are non-overlapping.

Now I'm going to be using the VPC structure that I've demonstrated earlier in this lesson, so we'll be assuming the usage of three availability zones plus a spare, and three application tiers plus a spare.

And this means that each VPC is broken down into a total of 16 subnets, and each of those subnets is a /20 subnet, which represents 4,091 IP addresses per subnet.

Now this might seem excessive, but we have to assume the highest possible growth potential for animals for life.

We've got the potential growth of the business, we've got the current situation with the environment, and the raising profile of animal welfare globally, so there is a potential that this business could grow rapidly.

This process might seem vague and abstract, but it's something that you'll need to do every time you create a well-designed environment in A.L.

US.

You'll consider the business needs, you'll avoid the ranges that you can't use, you'll allocate the remainder based on your business's physical or logical layout, and then you'll decide upon and create the VPC and subnet structure from there.

You'll always work either top-down or bottom-up.

You can start with the minimum subnet size that you need and work up, or start with the business requirements and work down.

When we start creating VPCs and services from now on in the course, we will be using this structure, and so I will be referring back to this lesson and that PDF document constantly, so you might want to save it somewhere safe or print it out, make sure you've got a copy handy because we will be referring back to it constantly as we're deciding upon our network topology throughout the course.

With that being said, though, that's everything I wanted to cover in this lesson.

I hope it's been useful and I hope it's been a little bit abstract, but I wanted to step you through the process that a real-world solutions architect would use when deciding on the size of subnets and the VPCs, as well as the different structure these network components would have in relation to each other's IEP plan.

But at this point, that is it with the abstract theory.

From this point onward in this section of the course, we're going to start talking about the technical aspects of AWS private networking, starting with VPCs and VPC subnets.

So go ahead, complete this video, and when you're ready, you can move on to next.

Welcome back.

In this lesson, I'm going to cover a topic that many courses don't bother with.

How to design a well-structured and scalable network inside AWS using a VPC.

Now, this lesson isn't about the technical side of VPC.

It's about how to design an IP plan for a business, which includes how to design an individual network within that plan, which when running in AWS means designing a VPC.

So let's get started and take a look because this is really important to understand, especially if you're looking to design real-world solutions or if you're looking to identify any problems or performance issues in exam questions.

Now, during this section of the course, you'll be learning about and creating a custom VPC, a private network inside AWS.

When creating a VPC, one of the first things you'll need to decide on is the IP range that the VPC will use, the VPC SIDA.

You can add more than one, but if you take architecture seriously, you need to know what range the VPC will use in advance.

Even if that range is made up of multiple smaller ranges, you need to think about this stuff in advance.

Deciding on an IP plan and VPC structure in advance is one of the most critically important things you will do as a solutions architect, because it's not easy to change later and it will cause you a world of pain if you don't get it right.

Now, when you start this design process, there are a few things that you need to keep in mind.

First, what size should the VPC be?

This influences how many things, how many services can fit into that VPC.

Each service has one or more IPs and they occupy the space inside a VPC.

Secondly, you need to consider all the networks that you'll use or that you'll need to interact with.

In the previous lesson, I mentioned that overlapping or duplicate ranges would make network communication difficult, so choosing widely at this stage is essential.

Be mindful about ranges that other VPCs use, ranges which are utilized in other cloud environments, on other on-premises networks, and even partners and vendors.

Try to avoid ranges which other parties use which you might need to interact with and be cautious.

If in doubt, assume the worst.

You should also aim to predict what could happen in the future.

What the situation is now is important, but we all know that things change, so consider what things could be like in the future.

You also need to consider the structure of the VPC.

For a given ID range that we allocate to a VPC, it will need to be broken down further.

Every IT network will have tiers, Web tier, Application tier and Database tier are three common examples, but there are more, and these will depend on your exact IT architecture.

Tears are things which separate application components and allow different security to be applied for example.

Modern IT systems also have different resiliency zones, known as Availability zones in AWS.

Networks are often split, and parts of that network are assigned to each of these zones.

These are my starting points for any systems design.

As you can see, it goes beyond the technical considerations, and rightfully so, a good solid infrastructure platform is just as much about a good design as it is about a good technical implementation.

So since this course is structured around a scenario, what do we know about the Animals for Life organization so far?

We know that the organization has three major offices, London, New York and Seattle.

That will be three IP address ranges which we know are required for our global network.

We don't know what those networks are yet, but as Solutions Architects we can find out by talking to the IT staff of the business.

We know that the organization have field workers who are distributed globally, and so they'll consume services from a range of locations.

But how will they connect to the business?

Will they access services via web apps?

Will they connect to the business networks using a virtual private network or VPN?

We don't know, but again, we can ask the question to get this information.

What we do know is that the business has three networks which already exist.

192.168.10.0/24 which is the business's on-premise network in Brisbane.

10.0.0.0/16 which is the network used by an existing AWS pilot.

And finally, 172.31.0.0/16 which is used in an existing Azure pilot.

These are all ranges our new AWS network design cannot use and also cannot overlap with.

We might need to access data in these networks, we might need to migrate data from these networks, or in the case of the on-premises network it will need to access our new AWS deployment.

So we have to avoid these three ranges.

And this information that we have here is our starting point, but we can obtain more by asking the business.

Based on what we already know, we have to avoid 192.168.10.0/24, we have to avoid 10.0.0.0/16, and we have to avoid 172.31.0.0/16.

These are confirmed networks that are already in use.

And let's also assume that we've contacted the business and identified that the other on-premises networks which are in use by the business, 192.168.15.0/24 is used by the London office, 192.168.20.0/24 is used by the New York office, and 192.168.25.0/24 is used by the Seattle office.

We've also received some disturbing news.

The vendor who previously helped Animals for Life with their Google Cloud approval concept cannot confirm which networks are in use in Google Cloud, but what they have told us is that the default range is 10.128.0.0/9, and this is a huge amount of IP address space.

It starts at 10.128.0.0 and runs all the way through to 10.255.255.255, and so we can't use any of that if we're trying to be safe, which we are.

So this list would be my starting point.

When I'm designing an IP addressing plan for this business, I would not use any of this IP address space.

Now I want you to take a moment, pause the video if needed, and make sure you understand why each of these ranges can't be used.

Start trying to become familiar with how the network address and the prefix map onto the range of addresses that the network uses.

You know that the IP address represents the start of that range.

Can you start to see how the prefix helps you understand the end of that range?

Now with the bottom example for Google, remember that a /8 is one fixed value for the first octet of the IP, and then anything else.

Google's default uses /9, which is half of that.

So it starts at 10.128 and uses the remainder of that 10.space, so 10.128 through to 10.255.

And also an interesting fact, the Azure network is using the same IP address range as the AWS default VPC users.

So 172.31.0.0, and that means that we can't use the default VPC for anything production, which is fine because as I talked about earlier in the course, as architects, where possible, we avoid using the default VPC.

So at this point, if this was a production process, if we were really designing this for a real organization, we'd be starting to get a picture of what to avoid.

So now it's time to focus on what to pick.

Now there is a limit on VPC sizing in AWS.

A VPC can be at the smallest /28 network, so that's 16 IP addresses in total.

And at most, it can be a /16 network, which is just over 65,000 IP addresses.

Now I do have a personal preference, which is to use networks in the 10 range, so 10.x.y.z.

And given the maximum VPC size, this means that each of these /16 networks in this range would be 10.1, 10.2, 10.3, all the way through to 10.255.

I also find it important to avoid common ranges.

In my experience, this is logically 10.0 because everybody uses that as a default, and 10.1 because as human beings, everybody picks that one to avoid 10.0.

I'd also avoid anything up to and including 10.10 to be safe, and just because I like base 2 numbers, I would suggest a starting point of 10.16.

With this starting point in mind, we need to start thinking about the IP plan for the animals for life business.

We need to consider the number of networks that the business will need, because we'll allocate these networks starting from this 10.16.10 range.

Now the way I normally determine how many ranges a business requires is I like to start thinking about how many AWS regions the business will operate in.

Be cautious here and think of the highest possible number of regions that a business could ever operate in, and then add a few as a buffer.

At this point, we're going to be pre-allocating things in our IP plan, so caution is the term of the day.

I suggest ensuring that you have at least two ranges which can be used in each region in each AWS account that your business uses.

For animals for life, we really don't yet know how many regions the business will be operating in, but we can make an educated guess and then add some buffer to protect us against any growth.

Let's assume that the maximum number of regions the business will use is three regions in the US, one in Europe, and one in Australia.

That's a total of five regions.

We want to have two ranges in each region, so that's a total of five times two, so 10 ranges.

And we also need to make sure that we've got enough for all of our AWS accounts, so I'm going to assume four AWS accounts.

That's a total number of ID ranges of two in each of five regions, so that's 10, and then that in each of four accounts, so that's a total of ideally 40 ID ranges.

So to summarise where we are, we're going to use the 10 range.

We're going to avoid 10.0 to 10.10 because they're far too common.

We're going to start at 10.16 because that's a nice, clean, based-to number, and we can't use 10.128 through to 10.255 because potentially that's used by Google Cloud.

So that gives us a range of possibilities from 10.16 to 10.127 inclusive, which we can use to create our networks.

And that's plenty.

Okay, so this is the end of part one of this lesson.

It's getting a little bit on the long side, and so I wanted to add a break.

So that's it.

I'm going to take a little bit of time to get to the end of part one.

So I'm going to take a little bit of time to get to the end of part one.

Welcome to this lesson where I'm going to be talking about S3 access points, which is a feature of S3, which improves the manageability of S3 buckets, especially when you have buckets which are used by many different teams or users, or when buckets store objects with a wide range of functions.

Now we have a good amount to cover, so let's just jump in and get started.

S3 access points simplify the process of managing access to S3 buckets and objects.

I want you to imagine an S3 bucket with billions of objects using many different prefixes.

Imagine this bucket is accessed by hundreds of different teams within business.

Now by default you would have one bucket with one really complex bucket policy.

It would be hard to manage and prone to errors.

Access points allow you to conceptually split this.

You can create many access points for a bucket and each of these can have different policies, so different access controls from a permissions perspective.

But also, each access point can be limited in terms of where they can be accessed from.

So for example, a VPC or the internet.

Every access point has its own endpoint address and these can be given to different teams.

So rather than using the default endpoint for S3 and accessing the bucket as a whole, users can use a specifically created access point, along with that specific endpoint address, and get access to part of that bucket or the whole bucket, but with certain restrictions.

Now you can create access points using either the console UI, or you can use the CLI or ABI using create-access-point.

And it's important that you remember this command.

Please try and remember create-access-point.

Now it's going to be easier for you to understand this architecture if we look at it visually.

Let's explore...

Or how everything fits together using an architecture diagram.

And we're going to use a typical example.

Let's say that we have an S3 bucket and this bucket stores sensitive health information for the animals for life organization.

Now this includes health information about the animals, but also the staff working for the business, such as medical conditions and any vaccination status.

Now we have three different sets of staff.

We have admin staff, field workers and vet staff.

The admin staff look after the organization's employees.

The field workers actually visit other countries doing wildlife studies and helping animals.

And then the vet staff look after the medical needs of any animals which the business takes care of.

Now in this example, the business are also using a VPC, with some instances and other resources performing data analysis functions.

If we weren't able to use access points, then we'd have to manage the bucket as a monolithic entity, managing a large and complex bucket policy to control access for identities within this account and potentially other accounts.

And this can become unwieldy very quickly.

One option that we have is to use access points.

And you can sexually think of these as mini-buckets or views on the bucket.

So we might have three access points for our three types of users and then one for the VPC.

Now each of these access points will have a unique DNS address for accessing it and it's this DNS address that we would give to our staff.

But more than this, each access point also has its own policy.

And you can think about this as functionally equivalent to a bucket policy, in that it controls access to the objects in a bucket when using that access point.

So this can control access to certain objects, prefixes or certain tags on objects.

And so it's super powerful.

So now we have a unique DNS name and a unique policy.

Essentially we have mini-buckets which are independently controlled.

And this makes it much easier to manage the different use cases for the main bucket and our staff can access it via the access points.

From the VPC side, access points can be set to only allow a VPC origin, which means that the access point is tied to a specific VPC.

This will need a VPC endpoint in the VPC and the two can be tied together so that the S3 endpoint in the VPC can be configured to only allow access via the S3 access point.

Now one really crucial thing to understand permissions-wise is that any permissions defined on an access point need to be also defined on the bucket policy.

So in this example, if the...

That's Stafford Grant.

And an access via an access point policy, then the same would need to also be granted via the bucket policy.

Now you can do delegation where on the bucket policy you grant wide open access via the access point.

So as long as the access point is used, any action on that bucket's objects is allowed.

And then you'll be fine, more granular control over access to objects in that bucket using the access point policies.

And that's a pretty common permissions architecture to make things simpler to manage.

Now I've included a link attached to this lesson with more details on permissions delegation together with some example access point policies.

You won't need this for the exam, but if you do want to do a bit of extra reading, do make sure you check out the links included with this lesson.

Now at this point, that's everything I wanted to cover.

You just need to have an overview of how this works.

At this point, that's everything.

So thanks for watching.

Go ahead and complete this video.

And when you're ready, I look forward to you joining me in the next.

Welcome back and in this lesson I want to quickly cover S3 access logs which is a feature available within S3 which you need to be aware of.

Now it's pretty simple to understand and so we're going to cover it pretty quickly.

So let's jump in and get started.

The concept of access logging is simple enough.

We have in this example a source bucket and a target bucket.

Now the source bucket is what we want to gain visibility up.

So we want to understand what types of accesses are occurring to this source bucket.

And the target bucket is where we want the logging to go.

To use the feature we have to enable logging on the source bucket and this can be done either using the console UI or using a put bucket logging operation using the CLI or the API.

Now logging is managed by a system known as the S3 log delivery group which reads the logging configuration which you set on the source bucket.

It's important at this point to understand that this is a best efforts process.

Either enabling this feature or making changes in configuration can take a few hours to take effect.

To use this feature you also need to give the log delivery group access to the target bucket and this is done using an ACL on the target bucket.

You add the S3 log delivery group giving it right access and this is how it can deliver the logs through the target bucket.

Now logs are delivered as log files and each file consists of a number of log records and these and new line are limited.

Each record consists of attributes such as date and time, the requester, the operation, status codes, error codes and much more.

If you've seen an Apache log file then these are very similar.

Now each attribute within a record is spaced unlimited so the records within a file are new line limited and the attributes within a record are spaced limited.

Now a single target bucket can be used for many source buckets and you can separate these easily using prefixes in the target bucket and this is something that's configured within the logging configuration that's set on the source bucket.

Access logging provides detailed information about the requests which are made to a source bucket and they're useful for many different applications, most commonly security functions and any access audits and it can also help you to understand the access patterns of your customer base and understand any charges on your S3 bill.

If you do use this feature then you need to personally manage the lifecycle or deletion of any of the log files.

This is not built into the product so you need to manage either the movement of these log files to different storage classes or deleting them after a certain amount of time.

Now that's it for the architecture.

If you're doing a course where you do need practical experience then I'll be following this up with a demo.

If not then this theory is all that you'll need.

At this point, thanks for watching.

Go ahead and complete this video and when you're ready I look forward to you joining me in the next.

Welcome to this lesson where I want to step through the event notification feature of S3.

This is a feature which allows you to create event notification configurations on a bucket.

So let's jump in and get started exploring the architecture of this feature and exactly how we can use it.

The feature is pretty simple to understand.

When enabled, a notification is generated when a certain thing occurs within a bucket.

And these can be delivered to different destinations, including SNS topics, SQSQs or LAN functions.

And this means that you can have event-driven processes which occur as a result of things happening within S3.

Now, there is different types of events supported.

For example, you can generate event notifications when objects are created, which means put, post, copy and when long multi-part upload operations complete.

Maybe you want to do something like take an image and add a watermark or do something crazy like generating a retro pixel art version of any images uploaded to a bucket.

You can create an event notification configuration, set it so that it triggers whenever an object is created, send this to some destination to process the image, and then you have an event-driven automated workflow.

You can also set event notifications to trigger on object deletion so you can match any type of deletion using the Star Wild card.

You can match delete operations or when delete markers are created.

And you might use this if you want an automated security system to react if any objects are removed from any of your S3 buckets.

You can also have it trigger for object restores, so if you have objects in S3, Glacier or Glacierd Part 5 and you perform a restore operation, you can be notified when it starts and completes.

And this can be useful if you want to notify customers or staff when a restore begins and ends.

Then finally, you can get notifications relating to replication.

So if operations miss the 15-minute threshold, if they're replicated after the threshold, when an object is no longer tracked, and even if an object fails replication, all of those things can generate an event notification using this product.

Now, visually, it looks like...

Like this.

We start with an S3 bucket and define... ...an event notification configuration.

The configuration matches events which we want to be notified of.

Overtide these events occur which, as I just talked about, include create events, delete events, restore events and replicate events.

And when these things occur, events are generated and sent to the destinations which you configure.

And at the time of creating this lesson, that's Lambda, SQSQs and SNS topics.

These events interact with those services.

Events are generated by the S3 service known as the S3 principle.

And so we need to also add resource policies onto each of those destination services allowing the S3 service to interact with them.

This could be in the form of an SQSQ policy or a Lambda resource policy.

And at the time of creating this lesson, the only way to modify Lambda resource policies is to use the CLI or the API.

Although this might change over time.

The events themselves are actually JSON objects.

And this is a cut-down version just to give you an idea of what they look like.

Now, there are bits missing, I know, but you'll get the general idea.

If you use Lambda, for example, this is received within the event structure and you'll need to pass that, extract the information that you need and then act on it.

Now, S3 event notifications are actually a relatively old feature and support only a limited set of things occurring on objects in S3 buckets and can only interact with a limited number of AWS services.

You can also use EventBridge, which supports more types of events and can integrate with a wider range of AWS services.

As always with AWS, you have a few different ways of doing the same thing.

Now, as a default, I would tend to lean towards using EventBridge unless you have a specific reason not to do so.

But I did want to make sure that you understand the event notification feature of S3.

Now, at this point, that's everything I wanted to cover.

So go ahead and complete this video.

And when you're ready, I look forward to you joining me in the next.

Welcome back in this lesson, I want to cover a really important feature of S3 in Glacier that you need to be aware of for the exam.

Now you don't need to understand the implementation just the architecture.

S3 Select and Glacier Select are ways that you can retrieve parts of objects rather than the entire object.

Now I expect that this will feature only in the minor way in the exam and it will be focused on architecture and features but I do want you to be aware of exactly how this feature works.

So let's jump in and step through this architecture and what benefits it provides.

Now you know by now that both S3 and Glacier are super scalable services.

You can use them both to store huge quantities of data.

S3 for example can store objects up to five terabytes in size and can store an infinite number of those objects.

Now often when you're interacting with objects inside S3 you intentionally want to interact with that full object so you might want to retrieve that full five terabyte object.

What's critical to understand as a solutions architect is that logically if you retrieve a five terabyte object then it takes time and it consumes that full five terabytes of transfer.

So if you're downloading a five terabyte object from S3 into your application then you consume five terabytes of data you're accessing that full object and it takes time.

Now you can filter this on the client side but this occurs after the figurative damage is done.

You've already consumed that capacity, you've already downloaded that data, filtering it at the client side just means throwing away the data that you don't need.

S3 and Glacia provide services which allow you to access partial objects so that's what's provided by the S3 select and Glacia select services and the way that you do this is that both services allow you to create a SQL like statement so cut down SQL statement.

So you create this, you supply it to that service and then the service uses this SQL like statement to select part of that object and this part and only this part is sent to the client in a pre-filtered way so you only consume the pre-filtered part of that object, the part that you select so it's faster and it's cheaper.

Now both S3 select and Glacia select allow you to operate on a number of file formats with this level of functionality so examples of this include comma separated values JSON can even use visa to compression comma separated values and for JSON so it's a really flexible service.

Now visually this is how it looks.

Now at the top we have the architecture without using S3 or Glacia select and at the bottom we have the architecture when we utilize these services so in both cases we have an application which stores its data on S3.

So when an application interacts with S3 to retrieve any data the entire object or stream of objects are delivered to the application so the application receives everything.

It can either accept it as a whole or it can perform its own filtering but the critical thing to understand is that any filtering performed by the application is performed inside the application.

It doesn't impact the cost or performance.

The only data which is filtered out is simply discarded but it's still billed for and it still takes time.

Now contrast this to using S3 select with the same architecture so we still have the same application that interacts with the same S3 bucket.

It places the filter point inside the S3 service itself and this allows us to use a SQL-like expression and provide this to the S3 select service.

The S3 select service can then use the SQL-like expression and it can apply this to the raw data in S3 so in effect it's taking the raw data filtering it down but this is occurring inside the S3 service so the data that the application receives is the pre-filtered data and this means that we can achieve faster speeds and a significant reduction in cost.

Once the filtering occurs before it's transferred to our application it means that we get substantial benefits both in speed and in cost and this is because the S3 service doesn't have to load all of the data and deliver it to the application.

We're applying this filter at the source, the source of the data which is the S3 service itself.

Now this is a feature which our applications will need to explicitly use but as a solutions architect it's a powerful feature that you need to understand to improve the levels of performance in any systems that you design.

Now at this point that's everything I wanted to cover on to keep it brief because it's a product that I only expect to feature in a very very minor way in the exam and I do want you to be aware of its existence.

So thanks for watching, go ahead, complete this video and when you're ready I look forward to you joining me in the next lesson.

Welcome back.

And in this lesson, I want to talk about S3 Object Lock.

Now, this is something which is really important to understand for the SysOps certification, but equally, if you're developing applications running in AWS or architecting solutions for AWS, you also need to have an awareness.

Now, we've got a fair amount to cover, so let's jump in and get started.

S3 Object Lock is actually a group of related features which I'll talk about in this lesson, but it's something that you enable on new S3 buckets.

If you want to turn on Object Lock for existing buckets, then you need to contact AWS support.

And this is probably going to change over time, but at the time of creating this lesson, this is the current situation.

Now, when you create a bucket with Object Lock enabled, versioning is also enabled on that bucket.

Once you create a bucket with Object Lock and enable it, you cannot disable Object Lock or suspend versioning on that bucket.

Object Lock implements a right once, read many architecture, and this is known as Worm.

It means that you can set it so that Object Versions once created can't be overwritten or deleted.

And just to reiterate, since this is a pretty important point, the feature requires versioning to be enabled on a bucket.

And because of this, it's actually individual Object Versions which are locked.

Now, when we talk about Object Lock, there are actually two ways it manages Object Retention.

Retention periods and legal holds.

An Object Version can have both of these, one or the other or none of them.

And it's really important for the exam and for the real world to think of these as two different things.

So, S3 Object Lock Retention and S3 Object Lock Legal Hold.

Now, just like with bucket default encryption settings, these can be defined on individual Object Versions, or you can define bucket defaults for all of the Object Lock features.

Now, this is just the feature at the high level.

Next, I want to quickly step through the key points of the different retention methods.

So, Retention Period and Legal Hold.

And we're going to start with Retention Period.

With the Retention Period style of object locking, when you create the Object Lock, you specify a Retention Period in days and/or years.

One year means the Retention Period will end one year from when it's applied.

Now, there are two modes of Retention Period Lock which you can apply.

And it's really, really important that you understand how these work and the differences between the two.

One, because it matters for the exam, and two, because if you get it wrong, it will cause a world of pain.

The first mode is Compliance Mode.

If you set a Retention Period on an object using Compliance Mode, it means that an Object Version cannot be deleted or overwritten for the duration of the Retention Period.

But, it also means that the Retention Period itself cannot be reduced and the Retention Mode cannot be adjusted during the Retention Period.

So, no changes at all to the Object Version or Retention Period settings.

And this even includes the account root user.

So, no identity in the account can make any changes to Object Versions, delete Object Versions, or change the Retention Settings until the Retention Period expires.

So, this is serious business.

This is the most strict form of Object Lock.

Don't set this unless you really want that Object Version to stay around in its current form until the Retention Period expires.

Now, you've used this mode as the name suggests for compliance reasons.

An example of this might be medical or financial data.

If you have compliance laws stating that you have to keep data, for example, for three years, with no exceptions, then this is the mode that you set.

Now, a less strict version of this is Governance Mode.

With this mode, you still set a Retention Period, and while active, the Object Version cannot be deleted or changed in any way.

But you can grant special permissions to allow these to be changed.

So, if you want a specific group of identities to be able to change settings and Object Versions, then you can provide them with the permission S3 colon Bypass Governance Retention.

And as long as they have that permission and they provide a header along with their request, which is X-AMZ-BYPASS-GOVERNANCE-RETENTION, then they can override the Governance Mode of Retention.

Now, an important point to understand is this last header, so X-AMZ-HIVEN, and then all the rest.

This is actually the default for the console UI.

And so, using the console UI, you have the S3 colon Bypass Governance Retention, you will be able to make changes to Governance Mode Retention Locks.

So, Governance Mode is useful for a few things.

One, if you want to prevent accidental deletion.

Two, if you have process reasons or governance reasons to keep Object Versions.

Or lastly, you might use it as a test of settings before picking the Compliance Mode.

So, that's Governance Mode.

These are both modes that can be used when using the Retention Period feature of S3 Object Locking.

So, please make sure you understand how they both work and the differences between the two before we finish with this lesson.

It's really, really critical that you understand.

Now, the last overview that I want to give is S3 Object Lock Legal Hold.

With this type, you don't actually set a Retention Period at all.

Instead, for an Object Version, you set Legal Hold to be on or off.

To repeat, there's no concept of retention.

This is a binary.

It's either on or off.

While Legal Hold is enabled on an Object Version, you can't delete or overwrite that specific Object Version.

An extra permission is required, which is S3 colon Put Object Legal Hold.

And this is required if you want to add or remove the Legal Hold feature.

And this type of Object Locking can be used to prevent accidental deletions of Object Versions, or for actual legal situations when you need to flag specific Object Versions as critical for a given case or a project.

Now, this point, let's take a moment to summarise and look visually at how all these different features work.

Let's start with Legal Hold.

We start with a normal Object and we upload it to a bucket with a setting Legal Hold Status to On.

And this means that the Object Version is locked until the Legal Hold is removed.

In this state, the Object Version can't be deleted or changed, but you can set the Legal Hold Status to Off, at which point normal Commissions apply and the Object Version can be deleted or replaced as required.

It's a binary.

It's either on or off, and that isn't the concept of Retention Period.

Next, we have the S3 Object Locks that use the Retention Period architecture.

First, we have Governance, so we put an Object into a bucket with a Lock Configuration of Governance and specify a Retention Period.

This creates a locked Object Version for a given number of days or years, and while it's in this state, it cannot be deleted or updated.

With Governance Mode, this can be bypassed if you have the permissions and specify the correct header.

And once again, this header is the default in the console, so you can adjust or remove the Object Lock or delete or replace the Object Version.

So the important thing to realise here is while an Object is locked for a given Retention Period using the Governance Mode, you can't make any changes to Object Versions or delete them, but you can be provided with the S3 colon bypass and Governance Retention Permission, and as long as you have that and specify the X-AMZ-VIPAS-GOVERNANCE-RETENTION-TRUEHEADER, then you can override the Governance Mode Object Lock during the Retention Period.

Then lastly, we have Compliance, which is the same architecture.

We upload an Object.

We specify Compliance Mode together with the Retention Period, and this creates an Object Version, which is locked for a certain period in days and years.

The difference though is that this can't be changed.

An Object Version can't be deleted or updated.

The Retention Period cannot be shortened.

The Compliance Model can't be changed to something else even by the account root user.

This is permanent.

Only once the Retention Period expires can the Object Version or the Retention Settings be updated.

And for all of these, they can be set on Object Versions or what defaults can be fine.

And that's the architecture of S3 Object Lock.

It's critical that you understand this.

If it takes a few watches at this lesson, then that's OK.

Make sure you understand it in detail, including how each type differs from the others.

And remember, they can be used in conjunction with each other, so the effects can overlap.

You might use Legal Hold together with either Governance or Compliance, and if you do, then the effects of this could overlap, so you need to understand all of this in detail.

But at this point, that's everything I wanted to cover in this lesson, so go ahead and complete the video.

I mean, you ready?

I look forward to you joining me in the next.

Welcome to this mini project where you're going to get the experience of creating S3 multi-region access points.

Now multi-region access points give you the ability to create a single S3 global endpoint and point this at multiple S3 buckets.

It's an effective way to use a single endpoint throughout requests to the closest S3 service.

Now in order to do this mini project you need to be logged in to an AWS account with admin permissions.

If you're using one of my courses then you should use the IAM admin user of the general AWS account which is the management account of the organization.

If you're not using my courses make sure you're using an identity with admin permissions.

You'll also need to select two different AWS regions before this mini project we're going to create S3 buckets in two regions.

I'm going to use AP, South East 2 or the Sydney region and CA Central 1 or the Canada region.

Now the first thing to do is to move to the S3 console so type S3 in the search box at the top and then open that in a new tab.

Once you're there we're going to create two buckets so first go ahead and click on create bucket.

Now we'll keep the bucket naming consistent so we'll use multi-heiven region -demo -heiven and then the region that you're in.

So in my case Sydney and then at the end I want you to append on a random number.

In my case 1-3-3-7.

Remember S3 bucket names need to be globally unique and this will ensure both of our buckets are.

Once you put the name make sure you set the region correctly everything else can be left as default apart from we need to enable bucket versioning.

So set this box under versioning to enable.

Now scroll to the bottom and click on create bucket.

Then we need to follow that same process again for the second bucket.

So click on create bucket.

Use the same bucket naming so multi-heiven region -demo -heiven and then the region name in this case Canada.

And make sure you append your random number and set the region.

Then scroll down, enable bucket versioning again and create the bucket.

Now once you've got these two buckets we're going to create the multi-region access point.

So click on multi-region access point on the menu on the left and click create multi-region access point.

For the name you can pick whatever you want it doesn't need to be globally unique only unique within an AWS account.

I'm going to pick really really critical cat data and then scroll down and add the buckets that you've just created.

These can't be added or edited after creation so we need to do it now.

Now we're going to add buckets.

Select the two buckets and then click on add buckets to confirm.

Once you've done that scroll down to the bottom and click create multi-region access point.

Now this process can take worst case up to 24 hours to complete but typically it creates much faster, generally around 10 to 30 minutes.

Now we do need this to be created before we continue.

So go ahead and pause the video, wait for the status on this to change to ready and then you're good to continue.

Okay so now that we've got this multi-region access point configured and it's ready to go.

Now that we've got this multi-region access point configured we need to configure replication between the two buckets because anyone using this multi-region access point will be directed to the closest S3 bucket and we need to make sure that the data in both matches.

So to do that go ahead and click on the multi-region access point name and go inside there and you'll see that the access point has an Amazon resource name as well as an alias.

Now you should probably note down the Amazon resource name because we might need it later on.

Once you've done that click on the replication and failover tab and you'll be able to see a graphical representation of any replication or failover configuration.

If we click on the replication tab you'll see there's no replication configured.

If we click on the failover tab you can see that we've got these two S3 buckets in different AWS regions configured as an active active failover configuration which means any requests made to this multi-region access point will be delivered to either of these S3 buckets as long as they're available.

Now we can click on one and click on edit routing status and configure it as passive which means it will only be used if no active buckets exist.

But in our case we want it to be active active so we'll leave both of these set to active.

Now we want to configure replication between the buckets so we're going to scroll down to replication rules and click create replication rule.

Now there are two templates available to start with, replicate objects amongst all specified buckets and replicate objects from one or more source buckets to one or more destination buckets.

Now which of these you pick depends on the architecture that you're using but because we have an active active configuration we want all the buckets to be the same.

So we're going to pick the replicate objects among all specified bucket template so this is replicating between every bucket and every other bucket.

Essentially it creates a set of buckets which contain exactly the same data all fronted by a multi-region access point.

So go ahead and make sure this template is selected and then click to select both of the buckets that you created.

In my case Sydney and Canada.

Once we've done that scroll down you can set whether you want the status to be enabled or disabled when created we're going to choose enable and you get to adjust the scope so you can either have it configured so that you can replicate objects using one or more filters or you can apply to all objects in the bucket.

Now we want to make sure the entire bucket is replicated so we're going to use apply to all objects in the bucket.

Now you're informed that an Ion role or roles will be generated based on your configuration and this will provide S3 with the permissions that it needs to replicate objects between the buckets.

Now this is informational we don't need to do anything so let's move on.

Now you're also told what encryption settings are used as well as the destination storage class so because of the template that we picked above we don't get to change the destination storage class and that's okay.

If we scroll down to the bottom we have additional replication options, we have replication time control which applies in SLA to the replication process, we have replication metrics and notifications to provide additional rich information and we can choose whether to replicate the lead markers and whether to replicate modifications.

Now for this mini project we're not going to use replication time control we don't need that level of SLA.

We are going to make sure that replication metrics and notifications is selected.

We don't want to replicate the lead markers and we do want to make sure that replica modifications sync is checked.

So we only want replication metrics and notifications and replica modifications sync.

So make sure that both of those are checked and then click on create replication roles.

Now at this point all the buckets within this multi-region access point are now replicating with each other.

In our case it's only the two, in my case it's Canada and Sydney.

So go ahead and click on close and we can see how this graphical representation has changed showing us that we now have two-way replication in my case between Sydney and Canada.

Now at this point we need to test out the multi-region access point and rather than having you configure your local command line interface we're going to do that with Cloud Shell.

Now what I want you to do is to go ahead and move to a different AWS region so not the same AWS regions that either of your buckets are created in.

What I do want you to do though is make a region close to one of your buckets.

Now I'm going to start off with Sydney and in order to test this I'm going to switch across to the Tokyo region which is relatively close to Sydney, at least from a global perspective.

So I'm going to click on the region drop down at the top and change it from Sydney to Tokyo.

And what's on there I'm going to click on this icon which starts the Cloud Shell.

If this is the first time you're using it in this region you'll probably get the welcome to AWS Cloud Shell notification.

Just either click on close or check this box and then click on close if you don't want to see this notification again.

Now all these commands that we're going to be running are in the instructions which are attached to this video.

The first thing that we're going to do is to create a test file that we're going to upload to S3.

We're going to do that using the DD command.

So we're going to have an input of /dev/urandom which just gives us a stream of random data.

And then for the output using the OF option we're going to create a file called test1.file.

This is going to have a block size of 1 meg and a count of 10 which means that it's going to create a 10 meg file called test1.file.

So run that command.

Now once you've done that just go back to the tab that you've got open to S3.

Scroll to the top, click on multi-region access points, check the access point that you've created and then just click on copy ARN to copy the ARN for this access point into your clipboard and then go back to Cloud Shell.

Next I'm going to do an LS making sure I just have a file created within Cloud Shell, I do.

And now I'm going to run this command so aws space S3, space CP for copy, space test1.file.

So this is the local file we created within Cloud Shell and then space and then S3 colon, double forward slash and then the ARN of the multi-region access point.

Now this command is going to copy the file that we created to this multi-region access point and this multi-region access point is going to direct us towards the closest S3 location that it serves which should be the bucket within the Sydney region.

So go ahead and run that command.

It's going to take a few moments to upload but when it does switch back to S3, go to buckets.

In my case I'm going to go to the Sydney bucket and I should see the file created in this bucket.

I do, that's good, so I'm going to go back to buckets and go to Canada and I don't yet see the object created in the Canada bucket and that's because replication can take a few minutes to replicate from the bucket where the object was stored through to the destination bucket.

If we just give this a few moments and keep hitting refresh, after a few moments we should see the same S1.file which has been replicated from the Sydney region through to the Canada region.

Now S3 replication isn't guaranteed to complete in a set time, especially if you haven't configured the replication time control option.

So it's fairly normal to see a small delay between when the object gets written to one bucket and when it's replicated to another.

Now we're going to try this with a different region, so go back to Cloud Shell and then click on the region drop down and we're going to pick a region which is close to our other bucket but not in the same region.

So the other bucket is created in the Canada region, so I'm going to pick a close region.

In this case I'm going to pick US East 2 which is in Ohio.

Once I've done that I'm going to go back to Cloud Shell.

Once I've done that I should be in Cloud Shell in a different AWS region.

So I'll need to recreate the test file.

In this case I'm going to call it test2.file and I'm going to use all the same options.

And again this command is contained in the instructions attached to this video.

So run that command and it will take a few moments to complete and then we're going to follow the same process.

We're going to upload this file to our S3 buckets using the multi-region access point.

So again just make sure you've got the ARN for the access point in your clipboard and then in the Cloud Shell type AWS space S3, space CP, space test2.file, space S3, colon, forward slash forward slash and then the ARN of the multi-region access point.

Again we're going to press enter, wait for this to upload and then check the buckets.

So run that command and then go back to S3, go to buckets.

I'm going to go to the bucket in Canada first and I'm going to hit refresh and we should see test2.file in this bucket which is good.

Then go to buckets and go to Sydney and we probably won't see that file just yet because it will take a few moments to replicate.

So keep hitting refresh and eventually you should see test2.file arrives in the S3 bucket.

Now this time we're going to run the same test but we're going to pick a region that is relatively in the middle between these two regions where our S3 buckets are created.

In my case I'm going to change the region to APSALV1 which is the Mumbai region.

And I'm going to follow exactly the same process and move to Cloud Shell.

I'm going to generate a new file so in this case it's test3.file.

Now before we upload this we're going to go to the S3 console, go to buckets and just make sure that we have a tab open for both the Canada and the Sydney bucket.

This test is going to be uploaded from a region that's in the middle of these two buckets and so we need to be able to quickly check which bucket receives the upload directly and which bucket receives the replicated copy.

So once you've got a tab open for both those buckets, go back to Cloud Shell and run this command.

So aws space S3 space CP, space test3.file, space S3//// and then the name of the S3 multi-region access point.

Once you've done that go ahead and run that command, it will take a few moments to upload and then move straight to the tabs that you've got open to the S3 buckets and refresh both of them.

If I look at Sydney it looks as though that receives the file straight away.

So this multi-region access point is directed us to Sydney.

If I move to Canada and hit refresh we can see that the file's not yet arrived so this is receiving the replicated copy.

And it does that after a few minutes we can see test3.file arrives in this bucket.

Now this one we're going to run a test to show the type of problem that can occur if you're using this type of global replicated configuration using multi-region access points.

What I want you to do is to open up two different Cloud Shells.

I want one Cloud Shell open in the Canada region so the region may have one of your buckets and I want the other Cloud Shell open in the AP Southeast two region, so the Sydney region.

So we've got one Cloud Shell open in the same region as each of our buckets.

Now once we've done that in one region, in my case I'm going to use the Canada region, I'm going to generate a file called test4.file.

The command is going to be the same as we've used in previous steps.

Then I'm going to copy the ARN of the multi-region access point into my clipboard and I'm going to type this command again in the Canada Cloud Shell.

So this command is going to copy this file to the multi-region access point.

Now because I'm doing this in the Canada region, it's almost guaranteed that the multi-region access point is going to direct us to the bucket which is also in the Canada region.

And so the bucket in this region is going to get the direct copy of the object and the bucket in the Sydney region is going to receive the replicated copy.

So I'm going to fully type out this command and before I run it I'm going to move it to the Sydney region and type out a slightly different command.

This command is also contained in the instructions attached to this lesson.

This command is AWS space S3 space CP space and then S3 colon forward slash forward slash and then the name of the multi-region access point and then forward slash test4.file and then space and then a period.

Now this command when we run it which we're not going to do yet is going to copy the test4.file object from the multi-region access point into our Cloud Shell.

Remember we haven't created this object yet.

Now because this Cloud Shell is in the Sydney region, the multi-region access point is almost certainly going to redirect us to the bucket in the Sydney region.

So let's move back to the Canada Cloud Shell, run this command which is probably going to copy this object into the Canada bucket, then move back to the Cloud Shell in the Sydney region and then run this command to copy the object from the multi-region access point into our Cloud Shell and we receive this error.

Now we're getting this error because the object test4.file doesn't exist within the bucket that the multi-region access point is directing us to.

So just to reiterate what we've done, we've created this object in the Canada region using the multi-region access point which is going to have created the object in the closest S3 resource which is the bucket also in the Canada region.

So the Canada region bucket has the direct copy of the object.

This will then take a few minutes to replicate to the Sydney bucket because we're using this reverse copy command in the Sydney region.

It's attempting to copy the test4.file object from the multi-region access point to our Cloud Shell but because this replication won't have occurred yet and because this is going to direct us at the bucket in the Sydney region, we get this object does not exist error.

And this is one of the issues you can experience when you're using multi-region access points in that there's a consistency lack.

You have to wait for the replication to occur before you can retrieve replicated objects from different buckets using the multi-region access point.

Now this process can be improved by using the RTC option when setting up replication but this does come with additional costs.

So this is something to keep in mind when you're using this type of architecture.

And if we keep running this command we'll see that it keeps failing over and over again until the replication process finishes and then we can copy down the object.

Now that is everything which I wanted to cover in this brief mini project in the multi-region access points.

At this point all that remains is for us to clean up the account and return it to the same state as it was at the start of this mini project.

So to do that we need to move back to the S3 console, go to multi-region access points, select the access point you created for this mini project and click delete.

And then you need to copy and paste the name and click delete to confirm.

Then we need to go ahead and move to buckets.

Select each of the buckets in turn, first click on empty and you need to copy and paste or type permanently delete and then click to confirm that empty process.

Once that's finished click on exit, select the same bucket again, this time click delete, copy and paste or type the bucket name and then click to confirm.

And then do the same process with the other buckets so first empty it, confirm that and then delete it and confirm that.

And then at that point all the billable elements created as part of this mini project have been deleted and we go to finish off this mini project.

So that's everything I wanted to cover, I hope it's been useful and I hope it's given you some practical experience of how to use multi-region access points together with S3 replication.

At this point that's everything to go ahead and complete this video and I hope you'll join me soon for another exciting mini project.

Welcome back.

In this brief demo lesson, you're going to get some experience working with S3 pre-signed URLs.

Now, as you learned in the theory lesson, a pre-signed URL is a type of URL which can be used to grant access to certain objects within an S3 bucket where the credentials for accessing that object are encoded on the URL.

I want to explain exactly what that means and how you can use it in the real world.

Now, to do that, we're not going to need to create any infrastructure using CloudFormation.

Instead, we're going to do it manually.

So first, I want you to make sure that you're logged in to the general AWS account as the I am admin user.

And as always, please make sure that you have the Northern Virginia region selected.

Assuming that's all good, go ahead and type S3 in the search box at the top and open that in a new tab.

We're going to create an S3 bucket within the general AWS account.

So go ahead and click on create bucket to be in that process.

Now, I want you to call the bucket animals for life media.

And because of the unique naming requirements of S3 buckets, you'll need to add some randomness onto the end of this name.

We'll need to select us - east - one for the region.

We can scroll past the bucket settings for block public access.

We don't need to enable bucket versioning.

We won't be using any form of encryption.

Just go ahead and click on create bucket.

Now I want you to go inside the bucket that you've created and click on the upload button to upload an object.

Now, at this point, we need to upload an image file to this bucket.

Any image file will do.

But I've included a sample one attached to this lesson if you don't have one available.

So click the link for the image download.

That will download an image called all5.jpeg.

Once that's downloaded to your local machine, click on add files, select the file, open it, and then go ahead and upload the file to the S3 bucket.

Once that's finished, you can go ahead and click on close.

And you'll see now that we've got our S3 bucket and one object uploaded to that bucket called all5.jpeg.

Next, go ahead and click on that object.

Now, I want to demonstrate how you can interact with this object in a number of different ways.

And the detail really matters here.

So we really need to be sure of exactly what differences there are between these different methods.

The first thing I want you to do is towards the top right of the screen, click on the open button.

You might get a pop-up notification if you do just allow pop-ups.

And what that's going to do is open a jpeg object in a new tab.

Now, I want to point out a number of really important points about how this object has been opened.

If you take a moment to review the URL that's been used to open this object, you'll note that a number of pieces of information have been specified on the URL, including AMZ-SecurityToken.

So essentially, a form of authentication has been provided on the URL which allows you to access this object.

Now, I want you to contrast this by what happens if we go back to the S3 bucket and just copy down the object URL into our clipboard and note how this does not contain any additional authentication information.

It's just the raw URL for this object.

So copy that into your clipboard and then open a new tab and paste that in.

What you're going to see is an access denied message.

And this makes sense because we're now attempting to access this object as an unauthenticated identity, just like any internet user, anyone browsing to this bucket would be doing.

We're not providing any authentication.

And so the only way that we can access the object in this way by not providing any authentication is if we made the bucket public and the bucket currently isn't public, which is why we're getting this access denied message.

Just to reiterate, we can access it by using the previous method because by opening it from within the console, the console is intelligent enough to add authentication information onto the URL which allows us to access this object.

So those are the differences between these two methods.

One is providing authentication and the other isn't.

So right now the only entity that is able to access any of the objects within this bucket is this AWS account and specifically the I am admin user.

The object has no public access.

So now what we're going to do is we're going to work with the scenario that you want to grant access to this object to somebody else for a limited amount of time.

So you don't want to provide the URL that includes authentication information, you want to provide a URL which allows access to that object for a limited amount of time.

That's important.

So we're going to generate a time limited pre-sign URL.

So go ahead and click on the Cloud Shell icon and this is going to open a Cloud Shell using the identity that you're currently logged in at.

So it's going to open a shell much like the one you'll see when you're connected to an EC2 instance but the credentials that you have in the shell are going to be the credentials of the identity that you're currently logged into AWS using in our case the I am admin user with a general AWS account.

You'll see a message saying preparing your terminal and then you'll be logged in to what looks like an EC2 instance prompt.

You can use the AWS CLI tool so I'm able to run an AWS space S3 space LS and this shell will interact with your current AWS account using your current credentials.

So in my case I'm able to see the Animals for Life media bucket which is in my general AWS account because I'm currently logged into this Cloud Shell using the credentials of my I am admin user.

Now to generate a pre-sign URL we have to use this command.

So use AWS to use the command line tools and then a space S3 because we're using the S3 service and then a space and then the word pre-sign because we want to generate a pre-signed URL and then a space and then we need the S3 URI to this object.

Now we can get that from the S3 console.

For every object you can see this unique URI so go ahead and copy this into your click mode go back to the Cloud Shell paste it in and then we'll need space and then we'll use double hyphen expires hyphen in and then a space and then we need to provide the number of seconds that this pre-signed URL will be valid for.

In this case we're going to use 180 which is a total of three minutes so this is in seconds so three minutes is 180 seconds.

So go ahead and press enter and this will generate you a unique pre-signed URL.

So go ahead and copy this into your click board and it is a really long URL so make sure that you get everything including HTTPS all the way to the end of this URL.

Copy that into your click board and then I want you to open a new tab and load that URL and there you go you can see this object loads up using this pre-signed URL.

Now this URL is valid only for 180 seconds.

To demonstrate that I'm going to skip ahead 180 seconds and demonstrate exactly what happens when this URL expires.

After 180 seconds when a next refresh we see this access denied page with a message request has expired.

So you can see how pre-signed URLs are a really effective way of granting access to objects within an S3 bucket for a limited amount of time.

Now there are a number of really interesting aspects to pre-signed URLs that you really need to understand as an architect, a developer or an engineer and I want to go through these really interesting aspects of pre-signed URLs before we finish up with this demo lesson.

Now first just to make everything easier to see I'm going to close down any of these tabs that we got open to this all five object.

I'm going to go back to the cloud shell and I'm going to generate a new pre-signed URL but this time I'm going to use a much larger expires in time.

So I'm going to press the up arrow to return to the previous command.

I'm going to delete this 180 second expiring time and instead I'm going to use 604,800 which is a pretty high number but this is something that we can pick so that we won't have any unintentional expires at the URL as part of this demo lesson.

So pick something crazily large just to make sure that it doesn't expire until we're ready.

So we're generating another URL and so this is an additional unique pre-signed URL.

So I'm going to select all of this URL and you need to do the same go ahead and copy that into your clipboard and open it in a new tab.

Now we can see that that pre-signed URL has opened.

Keep in mind that we've generated this using the identity that we're currently logged into AWS using.

So next what I'm going to do is move back to the AWS console and I'm going to click on the services drop down and move across to IAM.

So I'm going to open IAM up in a new tab.

I'm going to select the users option, select the IAM admin user, I'm going to add an inline policy to this user, select JSON.

Now attached to this lesson is another link for a file called deny s3.json.

Go ahead and click that link and then you'll need to get the contents of the file.

So copy all the contents of the file into your clipboard and then select all of this JSON and paste in the contents of that file and this is an explicit deny policy which denies our user so IAM admin any access to s3.

So this essentially prevents our IAM admin user any level of access to s3.

Go ahead and click on review policy for name, call it deny s3 and click on create policy and this will attach this as an inline policy to our IAM admin user.

Now I'm going to clear the screen within this cloud shell to make it easier to see and then I'm going to run an AWS space s3 space ls and press enter.

Now we get an access deny when accessing the s3 service because we just added an explicit deny onto the IAM admin user and remember the rule that applies to permissions deny allow deny and explicit deny always overrules everything else and so even though the IAM admin user has administrative permissions by applying this deny s3 policy which has an s3 explicit deny this wins so currently we have no access to s3.

Now let's return to the tab where we have this pre-signed url open.

So remember this pre-signed url was generated at the time when we had access to s3 so now let's refresh this page and now we get an access denied message and this is one of those interesting aspects of pre-signed urls.

When you generate a pre-signed url using an identity such as an IAM user that url has your access permissions so you generate a pre-signed url and anyone using that pre-signed url for the duration that it's active will be interacting with that one object on s3 as though they were using your identity.

Now if you adjust the permissions on your identity as we've just done by denying access to s3 it means that that pre-signed url will also have that deny s3 permissions and so this pre-signed url now no longer has any access to anything in s3 including the object that it was configured to provide access to so now let's go back to our cloud shell and what we're going to do now remember we are still denied access to s3.

Let's press the app arrow and move back to the command which we used to generate this pre-signed url.

Now let's regenerate another pre-signed url.

Now note how even though we are denied access to any part of s3 we can generate a pre-signed url which points at this one specific object all five dot jpeg so we're not prevented from generating a pre-signed url for something that we have no access to.

Now if we copy this into our clipboard move to the tab that we already have open and just replace this url with the pre-signed url that you just generated.

Note that we're still denied we're still denied because although we could generate this pre-signed url we've generated it using an identity which has no access to s3 so the next interesting fact to pre-signed urls that I want to demonstrate if we go back to the iam console and now we remove this deny s3 policy from our iam admin user now our iam admin user once again has access to s3 so if we go back to the cloud shell we can run natal us space s3 space ls and press enter and now we can access s3 again from this cloud shell remember the cloud shell is using commissions based on our iam admin user now let's go back to the pre-signed url and now if we refresh this now we can access this object again so I'm doing this to illustrate that when you generate a pre-signed url that pre-signed url is linked to the identity that generates it whatever the permissions are of that identity when the pre-signed url is used that is the permissions that that pre-signed url has so for the duration it always has the same permissions that the identity which generated it has at that very moment so that's something that you need to keep in mind now more interestingly is that you can actually generate a pre-signed url for a non-existent object there's nothing preventing that from occurring so if we go back to the cloud shell we press the up arrow and a number of times to bring up this pre-signed url command and this time we try to generate a pre-signed url for an object which doesn't exist so if I change this all 5 to all 1 3 3 7 and press enter it will generate a pre-signed url that pre-signed url will be valid to access an object called all 1 3 3 7 .jpeg inside this bucket but because there's no such object in that bucket if I try to use it I won't be able to do so so if I open that new invalid pre-signed url I'll get the message that the specified key does not exist but we can generate pre-signed urls for non-existent objects now one more interesting thing about pre-signed urls which I'm not going to demonstrate is if you generate a pre-signed url using temporary credentials you get by issuing a roll so for example if we logged into an ec2 instance which had an instance roll on that instance and then we generated a pre-signed url even if we set a huge expiry time so 604800 that pre-signed url would stop working when those temporary credentials for that roll also stopped working now it is possible to generate a pre-signed url from the console UI this is a relatively recent change from the object if you click on the object actions drop down you can click share with the pre-signed url you have to set the same settings so what you want the expiry to be in this particular case let's say 60 minutes and then I can go ahead and click on create pre-signed url now that's automatically copied into my clipboard and I can go ahead and move to a different tab paste that in and we can open the console UI generated pre-signed url so that's just an alternative way of doing the same process that we just used the cloud shell for now that's everything that I wanted to demonstrate in this demo lesson about pre-signed url and don't worry we're going to be talking about these more later in the course as well as looking at some alternatives what we need to do to tidy up this lesson is go back to the AWS console move to the S3 console and then just go ahead and empty and delete the bucket that you created so select animals for live media click on empty type or paste in permanently delete and then confirm or once that's successfully empty click on exit the bucket should still be selected and then go ahead and click on delete delete confirm that with the name of the bucket and then go ahead and delete the bucket and at this point we've cleaned up the account and the resources back in the same state as they were at the start of the lesson so I hope you've enjoyed this brief demo lesson go ahead complete the video and when you're ready I look forward to you joining me in the next video.

Welcome back and in this lesson I want to talk to you about S3 pre-signed URLs.

Pre-signed URLs are a way that you can give another person or application access to an object inside an S3 bucket using your credentials in a safe and secure way.

Let's have a look at how this works architecturally.

To illustrate how pre-signed URLs work, let's use this example architecture, an S3 bucket which doesn't have any public access configured.

So it's still in a default private configuration.

This means that in order to access the bucket or any resources in it an IAM user such as IAM admin would have to authenticate to AWS and be authorized to access the resource.

IAM admin would send credentials along with the access request, AWS would validate them at the time that the request is made and only then grant access to the object in S3.

Now one issue that we have is that because the bucket is private only authenticated users are able to access it.

Our masked man here has no way of providing authentication information to AWS because he doesn't have any and so any request that's unauthenticated would fail.

Now if giving the mystery user access to S3 is an essential requirement for the business then there are three common solutions at this point and none of them are ideal.

Number one is to give the mystery user an AWS identity.

Number two is to give the mystery user some AWS credentials to use or number three is to make the bucket or the object public.

Now none of these are ideal.

If the user only needs short term access to the bucket and objects the effort of supplying an identity seems excessive.

Just giving some credentials to the user appears on the surface to be a security risk and definitely if it's somebody else's credentials that's just bad practice.

Making the bucket public for anything which has sensitive data in it also appears to be less than ideal.

So one solution that AWS offers is to use pre-signed URLs and let's look at how this would work with an architecture example.

IAM admin is an AWS identity with some access rights granted via a permissions policy.

So IAM admin can make the request to S3 to generate a pre-signed URL.

She would need to provide her security credentials, specify a bucket name, an object key and an expiry date and time as well as indicate how the object would be accessed.

And S3 will create a pre-signed URL and return it.

This URL will have encoded inside it the details that IAM admin provided.

So which bucket is for which object is for it will be encoded with the fact that the IAM admin user generated it and it will be configured to expire at a certain date and time as requested by the IAM admin user.

The URL could then be passed to our mystery user and he or she could use it to access a specific object in the specific S3 bucket up until the point at which it expires.

When the pre-signed URL is used the holder of that URL is actually interacting with S3 as the person who generated it.

So for that specific object in that specific bucket until the timer expires our masked man is actually IAM admin.

Pre-signed URLs can be used for both downloads from S3 so get operations or uploads to S3 known as put operations.

So this type of architecture might be useful for the animals for life remote workers if they don't have access to AWS accounts or accessing from a secure location and just need to upload one specific object.

Now there's another type of architecture which commonly uses pre-signed URLs and I've mentioned this earlier in the course when I was talking about some benefits of S3.

I want you to think about a traditional application architecture.

On the left we have an application user with a laptop.

We've got an application server in the public cloud in the middle which hosts the application and for example say let's say this is a video processing application for wildlife videos that the animals for life organization manages.

We've learned that one of the real strengths of S3 is its ability to host the large media files and so the large wildlife video files have been migrated from the application server to a media S3 bucket.

But by doing this we've introduced a problem.

Previously the videos were hosted on the application server and it could control access to the video files.

If we host them on an S3 bucket then either every user needs an AWS identity so an IAM user to access the videos or we need to make the videos public so that the user running the web application can download them into her browser and neither of those are ideal.

Luckily pre-signed URLs offer a solution.

With pre-signed URLs we can keep the bucket private.

Then we can create an IAM user in our AWS account for this application.

Remember when I talked about choosing between an IAM user or a role I said that if you could visualize how many of a certain thing that would be using an identity then it's likely to suit an IAM user.

Well in this case we have one application and application service accounts are a common scenario where IAM users are used.

So for this example we create an IAM user for the application.

So when our application user interacts with a web application she makes a request and that's step one.

The request might be for a page of information on a bushfire which is currently happening in Australia which has a video file associated with it.

The application that's running on the server knows that it can directly return the information that the request is asking for but that the video is hosted on the private S3 bucket.

So it initiates a request to S3 asking for it to generate a pre-signed URL for that particular video object using the permissions of the IAM user that the application is using.

So IAM app one.

The S3 service then creates a URL which has encoded within it the authentication information for the IAM app one user.

The access to one object in the bucket for a short time limited basis maybe two to three hours.

And the S3 service then returns that to the application server and through to the end user.

The web application that's running on the user's laptop then uses this pre-signed URL to securely access the particular object that's stored on the media bucket.

Pre-signed URLs are often used when you offload media into S3 or as part of serverless architectures where access to a private S3 bucket needs to be controlled and you don't want to run thick application servers to broke at that access.

And we'll look at serverless architectures later in the course.

For now I just want to be sure that you understand what pre-signed URLs do and in the next lesson which is a demo lesson you're going to get the chance to experiment yourself and generate a pre-signed URL.

Pre-signed URLs can be used to access an object in a private S3 bucket with the access rights of the identity which generates them.

They're time limited and they encode all of the authentication information needed inside.

And they can be used to upload objects and download objects from S3.

Now I want to show you this in a demo because it's far easier to actually do it.

But before we do that I want to step through some exam power-ups.

There are a couple of really interesting facts about generating pre-signed URLs which might help you out in some exam questions.

Now first and this is a fairly odd behavior but you can create a pre-signed URL for an object that you have no access to.

The only requirement for generating a pre-signed URL is that you specify a particular object and an expiry date and time.

If you don't have access to that object you can still generate a pre-signed URL which also because it's linked to you will have no access to that object.

So there aren't many use cases where this is applicable but you do need to be aware that it is possible to generate a pre-signed URL when you don't have any access.

Now when you're using the URL, so when you utilize a URL and attempt to access an object, the permissions that you have access to match the identity that generated it.

And it's important to understand that it matches the permissions that the identity has right now.

So at the point when you use the URL, the URL has the same permissions as the identity that generated it has right now.

So if you get an access denied error when you attempt to use a pre-signed URL to access an object, it could be that the identity that generated that URL never had access or it could be that it simply doesn't have access right now.

And they're two very important nuances to understand about pre-signed URLs.

So when you're using a URL to access an object, it matches the current permissions of the identity that generated that URL.

Now that's fairly okay for an IM user.

As I demonstrated on the previous example of the application, generally you would create an IM user for the application and this IM user would have fairly static permissions.

So the permissions generally wouldn't change between when you created the URL and when your customer is using that URL within an application.

But don't generate pre-signed URLs based on an IM role.

You can in theory assume an IM role, which remember gives you temporary credentials, and then you can use those temporary credentials to generate a pre-signed URL.

A pre-signed URL can have a much longer validity period than those temporary credentials.

So those temporary credentials will generally expire well before a pre-signed URL does.

So if you generate a pre-signed URL using a role and then those temporary credentials expire, those credentials are no longer valid.

And so that URL will stop working.

And so it's almost never a good idea to generate a pre-signed URL using an IM role.

You should always use long-term identities.

So generally an IM user.

With that being said though, that is everything that I wanted to cover in this theory lesson.

So go ahead, complete the video, and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this demo lesson you're going to configure S3 cross-region replication.

So that's a replication of objects from one bucket to another bucket in different AWS regions.

And the scenario that we're going to be stepping through is where you as a systems engineer are looking to configure replication to allow disaster recovery of an S3 static website from one region to another AWS region.

So the first thing that I'll need you to do is to make sure that you're logged in to the general AWS account and you'll need to have the Northern Virginia region selected.

Assuming that's the case then go ahead and move to the S3 console because we're going to be creating our source and destination buckets.

Now if you see any notifications about updates to the UI or any additional functionality then just go ahead and close that down.

What we want you to do is to go ahead and click on create bucket.

Now we want to keep these names simple so that we can distinguish between source and destination bucket.

So for the source bucket we're going to start with source bucket and then we want you to use your initials in my case AC and then we want you to put a random number at the end and I'm going to attempt to use 1337.

For region for the source bucket let's use US - East - 1 and then scroll down past the block public access settings scroll down past bucket versioning and default encryption and then just create the bucket.

In my case it successfully created a bucket using this name and again because of the globally unique naming requirements of S3 buckets you need to make sure that you pick something different than me and different from other students.

Now the scenario is that we're using this bucket to host a static S3 website so we need to enable that functionality.

So go into source bucket, click on properties, scroll down all the way to the bottom and click on edit next to static website hosting and we need to enable static website hosting and for hosting type just make sure that you select host a static website.

Now we're going to use index.html for both the index document and for the error document.

So enter index.html in both of those boxes and once you've done that you can save those changes.

Now so that this bucket can host a static website we need to go to permissions and then we need to edit the block public access settings so click on edit and uncheck block all public access and once you've done so click on save changes.

You need to confirm that so follow the instructions and then confirm and this means that the bucket can be made public but in order to make it public we need to add a bucket policy.

So to edit the bucket policy scroll down below the block public access settings and then edit the bucket policy.

Now attached to this lesson is a demo files link.

I'll need you to click that link which will download a zip file.

Go ahead and extract that zip file which will create a folder and go inside that folder.

Containment in this folder are all the files which you'll need for this demo and inside that folder is a file called bucket_policy.json.

Go ahead and open that file so this is the file bucket_policy.

This is a bucket policy which allows any principal to use the star tile card to use the S3 get object action on this particular ARN.

Now this is a placeholder we need to update this placeholder so that it references any objects within our source bucket so I want you to copy this entire bucket policy into your clipboard, move back to the console, paste it in and once you've pasted it in go ahead and copy the bucket ARN for this bucket into your clipboard by clicking on this icon and then I want you to select the placeholder that you've just pasted in so straight after the first speech mark all the way up to before the forward slash so select this component of this placeholder ARN and paste it in the source bucket ARN so it should look like this.

It references ARN colon, ALUS colon, S3 and then colon, colon, colon and then the source bucket so whatever you've called your source bucket it should reference this so don't use what's on my screen, use your specific source bucket ARN and then you should have a forward slash star on the end.

That means that any anonymous or unauthenticated identity will be able to get any objects within this S3 bucket and that's what we want, we want this to be a public bucket.

So click on save changes to commit those changes and now this bucket is public, you get the warning under permissions overview that this bucket is public and under the bucket name at the top it will say publicly accessible.

Now once you've done that go back to the S3 console and we're going to create the destination bucket, we're going to follow exactly the same process so click on create bucket, this time we'll call it destination bucket and then again your initials and then ideally the same random number so in my case 1-3-3-7.

Go ahead and click on the region drop down and instead of picking US-EAST-1 this time we're going to use US-WEST-1 Now to save us some time we're going to uncheck this block all public access while we're creating the bucket so that we don't need to do that afterwards and we'll need to acknowledge that by checking this box, scroll all the way down to the bottom and then click on create bucket, then we're going to go into destination bucket, select properties, move down to the bottom and we'll need to enable static website hosting so click edit, enable, choose the option to host a static website and then just like before we're going to use index.html for both the index document and the error document so enter index.html in both of those boxes and then save those changes.

Then we'll need to go to permissions and edit the bucket policy so scroll down, click on edit for bucket policy you'll need to copy this template bucket policy into your clipboard, paste it into the policy box and again we need to replace this placeholder so copy the destination bucket ARN into your clipboard, select from after the speech mark through to before the forward slash paste that in and that will now reference any objects in the destination bucket and go ahead and click on save changes so now at this point we have the source and destination bucket, both of them are public, both of them are set to be static websites and neither of them have any objects inside that bucket.

The next step is to enable cross region replication from the source bucket through to the destination bucket so to do that click on the source bucket, click on the management tab, scroll down and we need to configure a replication rule so click on create replication rule, we're told that replication requires versioning to be enabled for the source bucket and we're given a convenient button to click to enable versioning on this bucket and that's what we're going to do so click on enable bucket versioning and this means this bucket can now be the source of this replication rule for replication rule name, I'm going to call it static website dr for disaster recovery we want it to be enabled straight away so make sure the status is set to enabled now we can limit this replication rule to have a certain scope, we can filter it based on prefix or tags and this allows us to replicate only part of this bucket, in our case we want to replicate the entire bucket so we're going to select the option that this rule applies to all objects in the bucket so check this option now that we've configured the source, so we've configured the bucket name, the source region and chosen the scope of this replication rule now we need to decide on the destination, now for the destination we can use a bucket within the same AWS account or we can specify a bucket within another AWS account, if we choose another AWS account then we need to worry about object ownership we need to make sure that the objects which are replicated into the destination account have the correct ownership because by default they will be owned by our account because we will be creating them we can specify as part of the replication rule that the destination account owns those objects but because we're using the same account that doesn't apply in this scenario so we're going to choose a bucket within this account, click on browse s3 and we're going to select the destination bucket that we just created so select the destination bucket that you created in the previous step and then click on choose path you're going to be informed that as well as replication requiring versioning on the source bucket it also is required on the destination bucket and again you're presented with a convenient button to enable versioning on this destination bucket so click on enable bucket versioning, just to confirm versioning is required on both the source and destination bucket when you're using s3 replication so that's the destination configured, we've picked the destination bucket and enabled versioning, so scroll down next we need to give this replication rule the permissions that it needs to interact with different AWS resources so this rule needs the ability to read from the source bucket and write those objects into the destination bucket so we need to give this replication rule an IAM role which provides those permissions so click in the choose IAM role drop down and select create new role you could select an existing one if you already had one which was preconfigured but for this demo lesson we've done so we're going to create a new role now you can enable the ability of s3 replication to support objects which are encrypted with AWS KMS you do have to select that option explicitly though when you're creating a replication rule and in our case we're not going to be replicating any encrypted objects using sse-kms so we can leave this unchecked I've talked about this in the theory lesson but you can also change the storage class as part of the replication process so generally you're replicating objects from a primary location or a source bucket through to a destination bucket and this is often part of disaster recovery scenarios where you often want cheaper storage at the destination end and so the default is to maintain the storage class that's used in the source bucket but you can override that and change the storage class which is used when storing those replicated objects in the destination bucket now we're not going to set that in this demonstration but you do have the option to do so now these are all relatively new features of s3 replication so s3 replication has been a feature available at AWS for some time but over time they've evolved and enhanced the feature sets available to the product so you're able to set replication time control or RTC and this imposes an SLA on the replication process so by enabling it you ensure that 99.99% of new objects are replicated within 15 minutes and this feature also provides access to replication metrics and notifications but do note that this does come at an additional cost because s3 versioning is used on both the source and destination buckets if you do delete an object in a source bucket then by default that deletion is not replicated through to the destination so by default s3 replication does not replicate delete markers and I've talked about delete markers elsewhere in the course by selecting this option you do replicate delete markers and that means that deletions are replicated from the source through to the destination for now though we're not going to select any of these just go ahead and click on save after a few moments you'll be presented with this option so the default s3 replication historically didn't replicate any objects which existed in a bucket before you enabled replication you're now offered the ability to replicate existing objects within a bucket so if you wanted to do this if you had a bucket with objects that already existed then you could replicate those as part of starting this process now our bucket is empty so we won't do this anyway so go ahead and select no do not replicate existing objects and click submit and at this point replication is enabled between our source bucket and our destination bucket so let's give it a try what we're going to do is click at the top here next to the source bucket name and this will take us to the top level of the source bucket and we're going to upload some objects so go ahead and click on upload and then click on add files at this point go ahead and locate the folder which you downloaded and extracted earlier in this demo lesson so the demo files folder so go inside that folder and inside here as well as the bucket policy template which you used earlier you'll see two folders website and website 2 I want you to expand the website and select both the aotm.jpg object and the index.html object select both of those and make sure you use the ones from the website folder not the website 2 folder and then click on open once they're selected for upload you need to scroll down to the bottom and click upload and now upload both of those objects to our source bucket once that's complete we can close down that dialog then I want you to click on properties we're going to move down to the bottom and we're going to copy the bucket website endpoint into our clipboard and then open that in a new tab so this is the source bucket website endpoint and this opens animalsforlife.org animal.monthformarch and this is a picture of my cat Winky so let's go back to the S3 console move to the top go back to the main S3 console and then move in to the destination bucket now don't be concerned if you see something slightly different at this point the timetaping for replicating objects from source to destination can vary wildly if you don't enable the replication SLA option in our case because these objects are relatively small they should be replicated fairly quickly now at this point I want you to go ahead and pause this video and wait until you see two objects in your destination bucket don't be alarmed if these objects take five or ten minutes to appear just wait pause the video wait for those objects to appear and then you're good to continue you'll see that we have both of the objects which we added to the source bucket they're both now stored in the destination bucket and that's because they've been replicated using S3 replication what we can do is go to properties, move all the way down to the bottom copy the destination bucket website endpoint into our clipboard and open that in a new tab so this is the destination static website endpoint and again we see the same website animalsforlife.org animal.monthformarch again my cat Winky so next we're going to move back to the main S3 console so click on Amazon S3 and then move back to the source bucket click on upload again and we're going to replace the objects that we just added so click on add files and then expand the website to folder and go ahead and upload both of these objects so we have the same name O T N dot JPEG and index dot HTML go ahead and upload both of these and make sure you're uploading to the source bucket so upload those you'll need to confirm that that'll take a few moments to complete once you see that the upload was successful you can close down this dialogue and then you should still have the tab open to the source bucket if you go to that tab and refresh you'll see now that we have animalsforlife.org animal.monthforapril and this is my cat Truffles if we go immediately to the destination bucket and hit refresh on this bucket you might still see the picture of Winky if the image doesn't change and you keep refreshing it just give it a few minutes because without the replication SLA then the time taken to replicate from source to destination can vary significantly in my case it took about three or four minutes of refreshers but now we see that the destination bucket has been updated with the new image again this is a picture of my cat Truffles so this has been a very simple example of cross region replication at this point I've got everything that I wanted to do in this demo lesson so all we need to do is to clean up the account and return it to the same state as it was at the start of the demo so close down both of these tabs and just return to the main S3 console select the destination bucket and click empty confirm this by following the instructions and then confirm click exit with the bucket still selected, delete the bucket again you need to follow the instructions to confirm deletion of this bucket so go ahead and do that and then we need to follow the same process for the source bucket so select it and empty it, confirm that, close down this dialogue and then delete the bucket and again you need to confirm that, follow those instructions and delete bucket then we need to click on services, move to the IAM console because remember S3 replication created an IAM role that was used before that replication so click on roles, locate the role which starts with S3 C R R and it will have the name of the source bucket as part of that role name so go ahead and select the correct role, make sure that you select the one which has the name of your source bucket and then delete that role and you need to confirm that process and that will delete the role and at this point that's the account back in the same state as it was at the start of this demo lesson so I hope you've enjoyed this demo and a simple example of how to use S3 replication specifically in this case cross region replication from a source to a destination bucket at this point that's everything that I wanted to do, I'll be talking about S3 replication elsewhere in the course in other associate courses and at the professional level so don't worry, you'll get plenty of opportunities to explore this specific ease of functionality available as part of S3 but at this point that's everything I wanted to do in this demo lesson go ahead, complete the video and when you're ready I'll look forward to you joining me in the next.

Welcome back.

In this lesson, I want to talk about S3 replication, the feature which allows you to configure the replication of objects between a source and destination S3 bucket.

Now there are two types of replication supported by S3.

The first type, which has been available for some time, is cross-region replication or CRR, and that allows the replication of objects from source buckets to one or more destination buckets in different AWS regions.

The second type of replication announced more recently is same-region replication or SRR, which as the name suggests is the same process, but where both the source and destination buckets are in the same AWS region.

Now the architecture for both types of replication is pretty simple to understand once you've seen it visually.

It only differs depending on whether the buckets are in the same AWS accounts or different AWS accounts.

Both types of replication support both, so the buckets could be in the same or different AWS accounts.

In both cases, replication configuration is applied to the source bucket.

The replication configuration configures S3 to replicate from the source bucket to a destination bucket, and it specifies a few important things.

The first is logically the destination bucket to use as part of that replication, and another thing that's configured in the replication configuration is an IAM role to use for the replication process.

The role is configured to allow the S3 service to assume it, so that's defined in its trust policy.

The role's permissions policy gives it the permission to read objects on the source bucket and permissions to replicate those objects to the destination bucket.

And this is how replication is configured between source and destination buckets, and of course that replication is encrypted.

Now the configuration does define a few other items, but I'll talk about them on the next screen for now, and just focusing on this basic architecture.

There is one crucial difference between replication which occurs in the same AWS accounts versus different AWS accounts.

Inside one account, both S3 buckets are owned by the same AWS account, so they both trust that same AWS account that they're in.

That means that they both trust IAM as a service, which means that they both trust the IAM role.

For the same account, that means that the IAM role automatically has access to the source and the destination buckets as long as the role's permission policy grants the access.

If you're configuring replication between different AWS accounts, though, that's nothing off.

The destination bucket, because it's in a different AWS account, doesn't trust the source account or the role that's used to replicate the bucket contents.

So in different accounts, remember that the role that's configured to perform the replication isn't by default trusted by the destination account because it's a separate AWS account.

So if you're configuring this replication between different accounts, there's also a requirement to add a bucket policy on the destination bucket, which allows the role in the source account to replicate objects into it.

So you're using a bucket policy, which is a resource policy, to define the role in a separate account can write or replicate objects into that bucket.

Once this configuration is applied, so either the top configuration is the same account or the bottom, if it's different accounts, then S3 can perform the replication.

Now let's quickly review some of the options available for replication configuration, so that might actually come in handy for you to know.

The first important option is what to replicate.

The default is to replicate an entire source bucket to a destination bucket, so all objects, all prefixes, and all packs.

You can, though, choose a subset of objects, so you can create a rule that has a filter, and the filter can filter objects by prefix or tags or a combination of both, and that can define exactly what objects are replicated from the source to the destination.

You can also select which storage class the objects in the destination bucket will use.

Now the default is to use the same class, but you can pick a cheaper class if this is going to be a secondary copy of data.

Remember when I talked about the storage classes that are available in S3, I talked about infrequent access or one-zone infrequent access classes, which could be used for secondary data.

So with secondary data, you're able to tolerate a lower level of resiliency, so we could use one-zone infrequent access for the destination bucket objects, and we can do that because we've always got this primary copy in the source bucket, so we can achieve better economies by using a lower cost storage class in the destination.

So remember this is the example, the default is to use the same storage class on the destination as is used on the source, but you can override that in the replication configuration.

Now you can also define the ownership of the objects in the destination bucket.

The default is that they will be owned by the same account as the source bucket.

Now this is fine if both buckets are inside the same account.

That will mean that objects in the destination bucket will be owned by the same as the source bucket, which is the same account, so that's all good.

However, if the buckets are in different accounts, then by default the objects inside the destination bucket will be owned by the source bucket account, and that could mean when you end up in a situation where the destination account can't read those objects because they're owned by a different AWS account.

So with this option you can override that and you can set it so that anything created in the destination bucket is owned by the destination account.

And lastly there's an extra feature that can be enabled called replication time control or RTC, and this is a feature which adds a guaranteed 15-minute replication SLA onto this process.

Now without this, it's a best efforts process, but RTC adds this SLA, it's a guaranteed level of predictability, and it even adds additional monitoring so you can see which objects are queued for replication.

So this is something that you would tend to only use if you've got a really strict set of requirements from your business, and make sure that the destination bucket and source buckets are in sync as closely as possible.

If you don't require this, if this is just performing backups or it's just for a personal project, or if the source and destination buckets aren't required to always be in sync within this 15-minute window, then it's probably not worth adding this feature.

It's something to keep in mind and be aware of for the exam.

If you do see any questions that mention 15 minutes for replication, then you know that you need this replication time control.

Now there are some considerations that you should be aware of, especially for the exam.

These will come up in the exam, so please pay attention and try to remember these points.

The first thing is that by default replication isn't retroactive.

You enable replication on a pair of buckets of source and destination, and only from that point onward are objects replicated from source to destination.

So if you enable replication on a bucket which already has objects, those objects will not be replicated.

And related to this, in order to enable replication on a bucket, both the source and destination bucket need to have versioning enabled.

You'll be allowed to enable versioning as part of the process of enabling replication, but it is a requirement to have it on, so a bucket cannot be enabled for replication without versioning.

Now you can use S3 batch replication to replicate existing objects, but this is something that you need to specifically configure.

If you don't, just remember by default replication is not retroactive.

Secondly, it is a one-way replication process only.

Objects are replicated from the source to the destination.

If you add objects manually in the destination, they will not be replicated back to the source.

This is not a bi-directional replication.

It's one-way only.

Now more recently, AWS did add the feature which allows you to add bi-directional replication, but just be aware that this is an additional setting which you need to configure.

By default, replication is one-way, and enabling bi-directional replication is something you need to specifically configure, so keep that in mind.

Now in terms of what does get replicated from source to destination, replication is capable of handling objects which are unencrypted, so if you don't have any encryption on an object, it's capable of handling objects which are encrypted using S3 and S3, and it's even capable of handling objects which are encrypted using S3 and KMS, but this is an extra piece of configuration that you'll need to enable.

So there's configuration and there's extra permissions which are required, because of course KMS is emolved.

Now more recently, AWS did add the ability to replicate objects encrypted with SSE-C, so that's server-side encryption with custom-managed keys, but this is a relatively recent addition.

Historically, SSE-C was incompatible with cross or same-region replication.

Replication also requires that the owner of the source bucket needs permissions on the objects which will replicate, so in most cases, if you create a bucket in an account and you add those objects, then the owner of the object will be the source account, but if you grant cross-account access to a bucket, if you add a resource policy allowing all the AWS accounts to create objects in a bucket, it's possible that the source bucket account will not own some of those objects, and the style of replication can only replicate objects where the source account owns those objects.

So keep that in mind, and the limitation is it will not replicate system events, so if any changes are made in the source bucket by lifecycle management, they will not be replicated to the destination bucket, so only user events are replicated, and in addition to that, it can't replicate any objects inside a bucket that are using the Belysia or Glacia Deep Archive storage classes.

Now that makes sense, because Glacia and Belysia Deep Archive, while they are shown as being inside an S3 bucket, you need to conceptually think of them as separate storage products, so they cannot be replicated using this process.

And then lastly, it's important to understand that by default, deletes are not replicated between buckets, so the adding of a delete marker, which is how object deletions are handled for a version-enabled bucket, by default, these delete markers are not replicated.

Now you can enable that, but you need to be aware that by default, this isn't enabled.

So one of the important things I need to make sure you're aware of in terms of replication is why you would use replication.

What are some of the scenarios that you'll use replication for?

So for the same region replication specifically, you might use this process for log aggregation, so if you've got multiple different S3 buckets which store logs for different systems, then you could use this to aggregate those logs into a single S3 bucket.

You might want to use the same region replication to configure some sort of synchronization between production and test accounts.

Maybe you want to replicate data from prod to test periodically, or maybe you want to replicate some testing data into your prod account.

This can be configured in either direction, but a very common use case for same region replication is this replication between different AWS accounts, different functions, so prod and test, or different functional teams within your business.

You might want to use same region replication to implement resilience if you have strict sovereignty requirements.

So there are companies in certain sectors which cannot have data leaving a specific AWS region because of sovereignty requirements, so you can have same region replication replicating between different buckets and different accounts, and then you have this account isolation for that data.

So having a separate account with separate logins isolated to make an audit team or a security team replicates it into that account, it provides this account level isolation.

Obviously, if you don't have those sovereignty requirements, then you can use cross region replication and use replication to implement global resilience improvements, so you can have backups of your data copied to different AWS regions to cope with large scale failure.

You can also replicate data into different regions to reduce latency.

So if you have, for example, a web application or your application loads data, then obviously it might be latency sensitive, so you can replicate data from one AWS region to another, so the customers in that remote region can access the bucket that's closest to them, and that reduces latency generally gives them better performance.

Now that is everything I want to cover in this video, so go ahead and complete the video, and when you're ready, I look forward to you joining me in the next.

Welcome back and in this lesson I want to talk about S3 lifecycle configuration.

Now you can create lifecycle rules on S3 buckets which can automatically transition or expire objects in the bucket.

There are great ways to optimize the cost for larger S3 buckets and so they're an important skill to have as a solutions architect, developer or operational engineer.

Let's jump in and explore exactly what features lifecycle configuration adds to S3 buckets.

A lifecycle configuration at its foundation is a set of rules which apply to a bucket.

These rules consist of actions which apply based on criteria.

So do X if Y is true.

These rules can apply to a whole bucket or they can apply to groups of objects in that bucket defined by prefix or tags.

The actions which are applied are one of two types.

Transition actions which change the storage class of whichever object or objects are affected.

So an example of this is that you could transition objects from S3 standard to say S3 infrequent access after 30 days.

And you do this if you're sure that the objects will be more infrequently accessed after that initial 30 day period.

You could then maybe transition the objects from infrequent access to S3 Glacier Depardive after 90 days.

But you're certain that they will be rarely if ever accessed after that point.

Now the other type of rule are expiration actions which can delete whatever object or object versions are affected.

So you might want to expire objects or versions entirely after a certain time period.

And this can be useful to keep buckets tidy.

Now both of these can work on versions as well if you have a version enabled bucket.

But this can become complex and so it's something that you need to carefully plan before implementing.

Life cycle configurations offer a way to automate the deletion of objects or object versions or change the storage class of objects to optimize costs over time.

And the important thing to understand is that these rules aren't based on access.

So you can't move objects between classes based on how frequently they're accessed.

This is something which is a different feature intelligent tearing does on your behalf.

Now you can combine these S3 life cycle actions to manage an object's complete life cycle.

For example suppose that objects you create have a well defined life cycle.

Initially the objects are frequently accessed over a period of 30 days.

Then objects are infrequently accessed for up to 90 days.

And after that the objects are no longer required.

In this scenario you can create an S3 life cycle rule in which you specify the initial transition action to S3 intelligent tearing, S3 standard IA or even S3 one zone IA storage.

Then you might have another transition action through to one of the glacier storage classes for archiving and then potentially even an expiration action when you're sure you no longer require the object.

As you move the objects from one storage class to another you save on the storage costs.

Now let's look visually at the life cycle process.

So with life cycle configurations specifically the transitions between different storage classes you can think of this as a waterfall.

So we start with all of the storage classes.

So we've got standard infrequent access, intelligent tearing, one zone infrequent access, glacier instant retrieval, glacier flexible retrieval and then glacier deep archive.

And it's important to understand that this transition process like a waterfall flows down.

So glacier flexible retrieval can transition into glacier deep archive.

Glacier instant retrieval can transition into glacier flexible and glacier deep archive.

One zone infrequent access can transition into glacier flexible or glacier deep archive.

Importantly not into glacier instant retrieval.

This is one of the exceptions you need to be aware of.

Intelligent tearing this one can transition into one zone infrequent access, glacier instant, glacier flexible and glacier deep archive.

Then we've got standard infrequent access which can transition into more classes still.

So intelligent tearing, one zone infrequent access, glacier instant, glacier flexible and glacier deep archive.

And then finally we've got S3 standard and this can transition into all of the other classes.

So transitioning flows down like a waterfall but generally you won't go directly.

Most of the life cycle configuration that I've been exposed to operates with multiple different stages as the access patterns of objects change over time.

If you do have any use cases though you can go directly between most of the different classes.

Whichever storage classes you do make use of transition can't happen in an upward direction only down.

And then finally there are some restrictions or considerations that you need to be aware of.

Firstly be careful when transitioning smaller objects from standard through to infrequent access, intelligent tearing or one zone infrequent access.

This is because of the minimums on those classes.

For larger collections of smaller objects you can end up with equal or more costs because of the minimums of these different storage classes.

Additionally this is important to understand for the exam.

There's a 30 day minimum period where an object needs to remain on S3 standard before then moving into infrequent access or one zone infrequent access.

So you can store objects when you upload them directly into standard IA and one zone IA and that's okay.

But if you first store that object in the S3 standard and then you want to transition that into standard IA or one zone IA then the object needs to have been in S3 standard for 30 days before you can life cycle transition that into either of those infrequent access tiers.

That's really important to understand.

You can always directly adjust the storage class of an object for the CLI or console UI.

But when you're using life cycle configuration an object needs to be in S3 standard for 30 days before it can be transitioned into standard infrequent access or one zone infrequent access.

And then finally this one is a little bit more obscure still.

If you want to create a single rule which transitions objects from standard through to infrequent access or one zone infrequent access you have to wait an additional 30 days before then transitioning those objects through to any of the glacier classes.

You can't have a single rule which moves something from S3 standard through to infrequent access classes and then soon in 30 days moves those same objects through to the glacier classes.

With a single rule the object has to be within standard infrequent access or one zone infrequent access for 30 days before that same rule can then move those objects through to any of the glacier classes.

You can have two rules which do that process without that 30 day gap but a single rule there has to be a 30 day period where the object is in the infrequent access tiers before moving into the glacier tiers.

Now depending on the course that you're watching there might be a demo coming up elsewhere in the course where you'll use life cycle rules.

If this doesn't apply to the course that you're on don't be alarmed if that isn't a demo.

In any case that's all I wanted to cover in this theory lesson so go ahead and complete the video and when you're ready we can move on to the next.

Welcome back, this is part two of this lesson.

We're going to continue immediately from the end of part one.

So let's get started.

Now let's talk about the first of the Glacier Storage Classes, S3 Glacier Instant Retrieval.

If I had to summarize this storage class, it's like S3 standard in frequent access, except it offers cheaper storage, more expensive retrieval costs, and longer minimums.

Standard IA is designed for when you need data instantly, but not very often, say once a month.

Glacier Instant Retrieval extends this, so data where you still want instant retrieval, but where you might only access it say once every quarter.

In line with this, it has a minimum storage duration charge of 90 days versus the 30 days of standard in frequent access.

This class is the next step along the path of access frequency, as the access frequency of objects decrease, you can move them gradually from standard, then to standard in frequent access, and then to Glacier Instant Retrieval.

The important thing to remember about this specific S3 Glacier class is that you still have instant access to your data.

There's no retrieval process required, you can still use it like S3 standard and S3 standard in frequent access.

It's just that it costs you more if you need to access the data, but less if you don't.

Now let's move on to the next type of S3 Glacier Storage Class.

And the next one I want to talk about is S3 Glacier Flexible Retrieval, and this storage class was formally known as S3 Glacier.

The name was changed when the previously discussed Instant Retrieval class was added to the lineup of storage classes available within S3.

So Glacier Flexible Retrieval has the same three availability zone architecture as S3 standard and S3 standard in frequent access.

It has the same durability characteristics of 11-9s, and at the time of creating this lesson, S3 Glacier Flexible Retrieval has a storage cost which is about one-sixth of the cost of S3 standard.

So it's really cost effective, but there are some serious trade-offs which you have to accept in order to make use of it.

For the exam, it's these trade-offs which you need to be fully aware of.

Conceptually, I want you to think of objects stored with the Glacier Flexible Retrieval class as cold objects.

They aren't warm, they aren't ready for use, and this will form a good knowledge anchor for the exam.

Now because they're cold, they aren't immediately available, they can't be made public.

Well, you can see these objects within an S3 bucket, they're now just a pointer to that object.

To get access to them, you need to perform a retrieval process.

That's a specific operation, a job which needs to be run to gain access to the objects.

Now you pay for this retrieval process.

When you retrieve objects from S3 Glacier Flexible Retrieval, they're stored in the S3 standard in frequent access storage class on a temporary basis.

You access them and then they're removed.

You can retrieve them permanently by changing the class back to one of the S3 ones, but this is a different process.

Now retrieval jobs come in three different types.

We have expedited, which generally results in data being available within one to five minutes, and this is the most expensive.

We've got standard where data is usually accessible in three to five hours, and then a low cost bulk option where data is available in between five and 12 hours.

So the faster the job type, the more expensive.

Now this means that S3 Glacier Flexible Retrieval has a first byte latency of minutes or hours, and that's really important to know for the exam.

So while it's really cheap, you have to be able to tolerate, you can't make the objects public anymore, either in the bucket or using static website hosting, and two, when you do access the objects, it's not an immediate process.

So you can see the object metadata in the bucket, but the data itself is in chilled storage, and you need to retrieve that data in order to access it.

Now S3 Glacier Flexible Retrieval has some other limits, so a 40 kb minimum available size and a 90 day minimum available duration.

For the exam, Glacier Flexible Retrieval is for situations where you need to store archival data where frequent or real-time access isn't needed.

For example, yearly access, and you're OK with minutes to hours for retrieval operations.

So it's one of the cheapest forms of storage in S3, as long as you can tolerate the characteristics of the storage class, but it's not the cheapest form of storage.

That honor goes to S3 Glacier Deep Archive.

Now S3 Glacier Deep Archive is much cheaper than the storage class we were just discussing.

In exchange for that, there are even more restrictions which you need to be able to tolerate.

Conceptually, where S3 Glacier Flexible Retrieval, which data in a chilled state, Glacier Deep Archive is data in a frozen state.

Objects have minimum, so 40 kb minimum available size and 180 day minimum available duration.

Like Glacier Flexible Retrieval, objects cannot be made publicly accessible.

Access to the data requires a retrieval job.

Just like Glacier Flexible Retrieval, the jobs temporarily restore to S3 standard and frequent access, but those retrieval jobs take longer.

Standard is 12 hours and bulk is up to 48 hours, so this is much longer than Glacier Flexible Retrieval, and that's the compromise that you agree to.

The storage is a lot cheaper in exchange for much longer restore times.

Glacier Deep Archive should be used for data which is archival, which rarely, if ever, needs to be accessed, and where hours or days is tolerable for the retrieval process.

So it's not really suited to primary system backups because of this restore time.

It's more suited for secondary long-term archival backups or data which comes under legal or regulatory requirements in terms of retention length.

Now this being said, there's one final type of storage class which I want to cover, and that's intelligent tearing.

Now intelligent tearing is different from all the other storage classes which I've talked about.

It's actually the storage class which contains five different storage tiers.

With intelligent tearing, when you move objects into this class, there are a range of ways that an object can be stored.

It can be stored within a frequent access tier or an infrequent access tier, or for objects which are accessed even less frequently, there's an archive instant access, archive access, or deep archived set of tiers.

You can think of the frequent access tier like S3 standard and the infrequent access tier like S3 standard infrequent access, and the archive tiers are the same price of performance as S3, Glacier, instant retrieval, and flexible retrieval.

And the deep archive tier is the same price of performance as Glacier Deep Archive.

Now unlike the other S3 storage classes, you don't have to worry about moving objects between tiers.

With intelligent tearing, the intelligent tearing system does this for you.

Let's say that we have an object, say a picture of whiskers which is initially kind of popular and then not popular, and then it goes super viral.

Well if you store this object using the intelligent tearing storage class, it would monitor the usage of the object.

When the object is in regular use, it would stay within the frequent access tier and would have the same costs as S3 standard.

If the object isn't accessed for 30 days, then it would be moved automatically into the infrequent tier where it would stay while being stored at a lower rate.

Now at this stage you could also add configuration, so based on a bucket, prefix or object tag, any objects which are accessed less frequently can be moved into the three archive tiers.

Now there's a 90 day minimum for archive instant access, and this is fully automatic.

Think of this as a cheaper version of infrequent access for objects which are accessed even less frequently.

Crucially this tier, so archive instant access, still gives you access to the data automatically as and when you need it, just like infrequent access.

In addition to this, there are two more entirely optional tiers, archive access and deep archive.

And these can be configured so that objects move into them when they haven't been accessed for 98 to 270 days for archive access, or 180 through to 730 days for deep archive.

Now these are entirely optional, and it's worth mentioning that when objects are moved into these tiers, getting them back isn't immediate.

There's a retrieval time to bring them back, so only use these tiers when your application can tolerate asynchronous access patterns.

So archive instant access requires no application or system changes, it's just another tier for less frequently accessed objects with a lower cost.

Archive access and deep archive changes things, your applications must support these tiers because retrieving objects requires specific API calls.

Now if objects do stay in infrequent access or archive instant access, when the objects become super viral in access, these will be moved back to frequent access automatically with no retrieval charges.

Intelligent tiering has a monitoring and automation cost per 1000 objects instead of the retrieval cost.

So essentially the system manages the movement of data between these tiers automatically without any penalty for this management fee.

The cost of the tiers are the same as the base S3 tiers, standard and infrequent access, there's just the management fee on top.

So it's more flexible than S3 standard and S3 infrequent access, but it's more expensive because of the management fee.

Now intelligent tiering is designed for long-lived data where the usage is...

[Sounds of S3 storage] Changing or unknown, if the usage is static either frequently accessed or infrequently accessed, then you're better using the direct S3 storage class, either standard or infrequent access.

Intelligent tiering is only good if you have data where the pattern changes or you don't know it.

Now with that being said, that's all of the S3 storage classes which I want to cover.

That's at least enough technical information and context which you'll need for the exam and to get started in the real world.

So go ahead and complete the video and when you're ready, I look forward to you joining me in the next.

Welcome back.

In this lesson I want to cover S3 object storage classes.

Now this is something which is equally as important at the associate and the professional level.

You need to understand the costs relative to each other, the technical features and compromises, as well as the types of situations where you would and wouldn't use each of the storage classes.

Now we've got a lot to cover so let's jump in and get started.

The default storage class available within S3 is known as S3 Standard.

So with S3 Standard, when Bob stores his captures on S3 using the S3 ABI, the objects are stored across at least three availability zones.

And this level of replication means that S3 Standard is able to cope with multiple availability zone failure, while still safeguarding data.

So start with this as a foundation when comparing other storage classes, because this is a massively important part of the choice between different S3 storage classes.

Now this level of replication means that S3 Standard provides 11 nines of durability, and this means if you store 10 million objects within an S3 bucket, then on average you might lose one object every 10,000 years.

The replication uses MD5 checksums together with cyclic redundancy checks known as CRCs to detect and resolve any data issues.

Now when objects which are uploaded to S3 have been stored durably, S3 responds with HTTP 1.1 200 OK status.

This is important to remember for the-- [Pause] The exam, if you see this status, if S3 responds with a 200 code, then you know that your data has been stored durably within the product.

With S3 Standard, there are a number of components to how you'll build for the product.

You'll build a gigabyte per month fee for data stored within S3.

A dollar per gigabyte charge per transfer of data out of S3 and transfer into S3 is free, and then finally you have a price per 1,000 requests made to the product.

There are no specific retrieval fees, no minimum duration for objects stored, and no minimum object sizes.

Now this isn't true for the other storage classes, so this is something to focus on as a solutions architect and in the exam.

With S3 Standard, you aren't penalized in any way.

You don't get any discounts, but it's the most balanced class of storage when you look at the dollar cost versus the features and compromises.

Now S3 Standard makes data accessible immediately.

It has a first byte latency of milliseconds, and this means that when data is requested, it's available within milliseconds, and objects will be made publicly available.

This is either using S3 permissions or...

Or if you enable static website hosting and make all the contents of the bucket available to the public internet, if you're doing that, then S3 Standard supports both of these access architectures.

So for the exam, the critical point to remember is that S3 Standard should be used for frequently accessed data, which is important and non-replaceable.

It should be your default, and you should only investigate moving through the storage classes when you have a specific reason to do so.

Now let's move on and look at another storage class available within S3, and the next class I want to cover is S3 Standard in... [silence] which can be easily replaced.

So this means things like replica copies, if you're using same or cross region replication, then you can use this class for your replicated copy, or if you're generating intermediate data that you can afford to lose, then this storage class offers great value.

Don't use this for your only copy of data, because it's too risky.

Don't use this for critical data, because it's also too risky.

Don't use this for data which is frequently accessed, frequently changed or temporary data, because you'll be penalized by the duration and size minimums that this storage class is affected by.

Okay, so this is the end of part one of this lesson.

It was getting a little bit on the long side and I wanted to give you the opportunity to take a small break, maybe stretch your legs or make a coffee.

Part two will continue immediately from this point.

Welcome back and in this video I want to talk about S3 bucket keys which are a way to help S3 scale and reduce costs when using KMS encryption.

Let's jump in and take a look.

So let's look at a pretty typical architecture.

We have S3 in the middle, we have KMS on the right and inside we have a KMS key, the default S3 service key for this region which is named AWS/S3.

Then on the left we have a user Bob who's looking to upload some objects to this S3 bucket using KMS encryption.

Within S3 when you use KMS each object which is put into a bucket uses a unique data encryption key or DEK.

So let's have a look at how that works.

So when Bob begins his first PUT operation when the object is arriving in the bucket a call is made to KMS which uses the KMS key to generate a data encryption key unique to this object.

The object is encrypted and then the object and the unique data encryption key are stored side by side on S3.

Each object stored on S3 uses a unique data encryption key which is a single call to KMS to generate that data encryption key.

This means that for every single object that Bob uploads it needs a single unique call to KMS to generate a data encryption key to return that data encryption key to S3, use that key to encrypt the object and then store the two side by side.

On screen we have three individual PUTs.

But imagine if this was 30 or 300 or 300,000 every second.

This presents us with a serious problem.

KMS has a cost.

It means that using SSE-KMS carries an ever increasing cost which goes up based on the number of objects that you put into an S3 bucket.

And perhaps more of a problem is that there are throttling issues.

The generated data encryption key operation can only be run either 5,500, 10,000 or 50,000 times per second and this is shared across regions.

Now this exact number depends on which regions you use but this effectively places a limit on how often a single KMS key can be used to generate data encryption keys which limits the amount of PUTs that you can do to S3 every second.

And this is where bucket keys improve the situation.

So let's look at how.

So with bucket keys the architecture changes a little.

We have the same basic architecture but instead of the KMS key being used to generate each individual data encryption key, instead it's used to generate a time limited bucket key and conceptually this is given to the bucket.

This is then used for a period of time to generate any data encryption keys within the bucket for individual object encryption operations.

And this essentially offloads the work from KMS to S3.

It reduces the number of KMS API calls so reduces the cost and increases scalability.

Now it's worth noting that this is not retroactive.

It only affects objects and the object encryption process after it's enabled on a bucket.

So this is a great way that you can continue to use KMS for encryption with S3 but offload some of the intensive processing from KMS onto S3 reducing costs and improving scalability.

Now there are some things that you do need to keep in mind when you're using S3 bucket keys.

First, after you enable an S3 bucket key, if you're using CloudTrail to look at KMS logs, then those logs are going to show the bucket ARN instead of your object ARN.

Now additionally, because you're offloading a lot of the work from KMS to S3, you're going to see fewer CloudTrail events for KMS in those logs.

So that's logically offloading the work from KMS to S3 and instead of KMS keys being used to encrypt individual objects, they're used to generate the bucket key.

And so you're going to see the bucket in the logs not the object.

So keep that in mind.

Book keys also work with same region replication and cross region replication.

There are some nuances you need to keep in mind generally when S3 replicates an encrypted object.

It generally preserves the encryption settings of that encrypted object.

So the encrypted object in the destination bucket generally uses the same settings as the encrypted object in the source bucket.

Now if you're replicating a plain text object, so something that's not encrypted and you're replicating that through to a destination bucket which uses default encryption or an S3 bucket key, then S3 encrypts that object on its way through to the destination with the destination bucket's configuration.

And it's worth noting that this can result in e-tag changes between the source and the destination.

Now make sure that I include a link attached to this video which details all of these nuanced features when you're using S3 bucket keys together with same or cross region replication.

It's beyond the scope of this video, but it might be useful for the exam and the real world to be aware of these nuanced features and requirements as you're using the product.

Now with that being said, that is everything that I wanted to cover in this video.

So go ahead and complete the video and when you're ready, I'll look forward to you joining me in the next.

Welcome back and in this demo lesson I just want to give you the opportunity to gain some practical experience of how S3 handles encryption.

So what we're going to do is create an S3 bucket and into that bucket we're going to put a number of objects and for each one we're going to utilize a different type of server-side encryption.

So we'll be uploading one object using SSE-S3 so S3 managed encryption and then one object using SSE-KMS which will utilize KMS for key management.

So once we've uploaded those objects we'll experiment with some permissions changes just to see how each of these different encryption types work.

So let's get started.

Now the first thing you'll need to check is that you're logged into the IAM admin user of the general AWS account and you need to have the Northern Virginia region selected.

Then let's move to S3 so click in the services drop-down type S3 and then open that in a new tab and then click to move to the S3 console.

Now I want you to go ahead and click on create bucket and then just create a bucket called catpicks and then put some random text on the end.

You should use something different and something that's unique to you.

Leave the region set as US East Northern Virginia scroll all the way down to the bottom and click on create bucket.

Next change across to the key management service console that will either be in your history or you'll have to type it in the services drop-down.

Once you hear go ahead and click on create key, pick symmetric key then expand advanced options, make sure KMS is selected as well as single region key.

Click next.

The alias we'll be using is catpicks, so type catpicks and click next.

Don't select anything for define key administrative permissions.

We'll not set any permissions on the key policy this time so just click next and then on this screen define key usage permissions just click next again without selecting anything.

So the key policy that's going to be created for this KMS key only trusts the account so only trust the account user of this specific AWS account and that's what we want.

So go ahead and click on finish.

At this point move back across to the S3 console, go into the catpicks bucket that you just created and I'll let you to go ahead and download the file that's attached to this lesson and then extract that file and inside the resultant folder are some objects that you're going to be uploading to this S3 bucket.

So go ahead and do that and then click on upload and files then locate the folder that you just extracted and go into it and you should see three images in that folder default -merlin.jpg, sse -kms -ginny.jpg and sse -s3 -dwees.jpg.

So that's good so the first one that we want to upload need to do these one by one because we're going to be configuring the encryption type to use.

So the first one is sse -s3 -dwees.jpg so select that and click on open, expand properties and then scroll down to serve aside encryption and it's here where you can specify to accept the bucket defaults by not specifying an encryption key or you can specify an encryption key.

Now when you pick to specify an encryption key you're again offered the ability to use the bucket default settings for encryption or you can override bucket settings and choose between two different types either Amazon s3 key which is sse -s3 or AWS key management service which is sse -kms.

Now for this upload we're going to use sse -s3 so Amazon s3 managed keys so select this option scroll all the way down to the bottom and click upload.

Wait for this upload process to complete and then click on close.

Now we're going to follow that same process again so click on upload again, add files.

This time sse -kms -ginny.

So select that click on open, expand properties and then scroll down to serve aside encryption then click specify an encryption key override bucket settings for default encryption and this time we're going to use sse -kms so select that and then select choose from your AWS kms keys and then you can either use the AWS managed key so this is the service default key so AWS forward slash s3 or you can choose your own kms key to use for this object.

What I'm going to do first is select this AWS managed key so the default key for this service and scroll all the way down to the bottom and click on upload.

Wait for this upload process to complete and then click on close so that will encrypt the object using this default s3 AWS managed kms key that's now inside kms.

I wanted to do that just to demonstrate how it automatically creates it so now let's go ahead and re-upload this object so click on upload, add files, select sse -kms -ginny.jpg and click on open, scroll down, expand properties, scroll down again, for service side encryption select specify an encryption key, select override bucket settings for default encryption, pick AWS key management service key so sse -kms, select choose from your AWS kms keys, click in the drop down and then select captics that we created earlier.

Once you've done that scroll all the way down to the bottom click on upload, wait for that process to complete and then click on close.

Now at this point we've got two different objects in this bucket and we're going to open both of these.

We're going to start with sse -s3 -dwees so let's click on it and then click on open and that works and then let's try the other object sse -kms -ginny so click on that and click on open and that also opens okay because IAM admin is a full administrator of the entire AWS account so that includes s3 and all the services including KMS.

Next what we're going to do is apply a deny policy on the IAM admin user which prevents us from using KMS so we'll stay as being a full account administrator and a full s3 administrator but we're going to block off the KMS service entirely and I want to demonstrate exactly what that does to our ability to open these three objects.

So click on services and either open the IAM console from the history or type it in the find services box and then click it.

Once we're here click on users select your IAM admin user, click add permissions and then create inline policy, click on the JSON tab and then delete the skeleton template that's in this box and then paste in the contents of the deny kms.json file and this is contained in the folder you extracted from this lessons download link and it's also attached to this lesson.

This is what it should look like the effect is to deny it denies any actions KMS call on star so any KMS actions on all resources so essentially this blocks off the entire KMS service for this user.

So go ahead and click on review policy call it deny KMS and click on create policy and this now means that IAM admin can no longer access KMS so now if we go back to the s3 console go inside the capex bucket we should still be able to open sse-s3 dweez.jpeg object.

If we click that click on open because this is encrypted using sse-s3 and this is completely internal to the s3 product we should have no problems owning this object because we have the permissions inside s3 to do anything in s3 but something different happens if we try to open the sse-kms-ginning object.

Now just to explain what will occur when I click on this open link s3 will then have to liaise with KMS and get KMS to decrypt the data encryption key that encrypts this object so we need to retrieve the encrypted data encryption key for this object and request that KMS decrypts it.

Now if that worked we'd get back the plain text version of that key and we would use it to decrypt this object and it would open up in a tab without any issues because we've got full rights over s3 we have permission to do almost all of that process but what we don't have permission to do is to actually get KMS to decrypt this encrypted data encryption key.

We don't have that permission because we just added a deny policy to the IAM admin user and as we know by now deny allow deny deny always wins and explicit deny always overrules everything else.

So when I click on open and s3 retrieves this encrypted data encryption key and gives it to KMS and says please decrypt this and give me the plain text back KMS is going to refuse and what we see when we do that is we get an access deny.

So now we've implemented this role separation so even though we have full s3 admin rights so if I went back to this bucket and I clicked on the sse-kms-ginning file and deleted it I would be able to delete that object because I have full control over s3 but I can't open it because I've prevented us accessing KMS and that's how we implement role separation.

So sse-kms is definitely the encryption type that you want to use if you've got these really extensive regulations or any security requirements around key control.

So let's go ahead and just remove that restriction so just go back to the IAM console.

I just want to do this before we forget and have problems later in the course.

Click on users click on IAM admin check the box next to deny KMS and then click remove and confirm that removal and that will allow us to access KMS again.

We can verify that by moving to the KMS console and we can bring up this list which proves that we've got some access to KMS again so that's good.

Now if we just go ahead and click on the AWS managed keys option on the left here this is where you'll be able to see this default encryption key that's used when you upload an object using sse-kms-incription but don't pick a particular key so this is now the default.

Now if we open this because it's an AWS managed key we don't have the ability to set any key rotation we can see the key policy here but we can't make any changes to it this is set by AWS when it creates it so that it only allows accesses from s3 so this is a fixed key policy but we can't control anything about this key.

Now contrast that with the customer managed keys that we've got and if we go into cat pics this is the key that we created now we can edit the key policy we could switch to policy view and make changes we've got the ability to control the key rotation so if you face any exam questions where you need to fully manage the keys that are used as part of the s3 encryption process then you've got to use sse-kms.

Now if we just return to the s3 console there's just one more thing that I want to demonstrate go into the cat pics bucket again click on properties locate default encryption and then click on edit and this is where you get the option to specify the default encryption to use for this bucket.

Now again this isn't a restriction this does not prevent anyone uploading objects to the bucket using a different type of encryption all it does is specify what the default is if the upload itself does not specify an encryption method so we could select Amazon s3 key which is sse-s3 and you might also see this referred to elsewhere as AES 256 it's also known by that name but we could also select the AWS key management service key known as sse-kms and this is where we can either choose to use the default key or pick a customer managed key that we want to use as the default for the bucket.

So let's just demonstrate that go ahead and select the cat pics key to use for this bucket then scroll down and click on save changes and that will set the defaults for the bucket and we can demonstrate that let's go ahead and click on the objects tab and we're going to upload a new object to this bucket so click on upload add files then go ahead and select the default hyphen Merlin object and click open scroll down click on upload and even though we didn't pick a particular encryption method for this object it's going to use the default settings that we picked for the bucket and now we can see that default hyphen Merlin.jpg object has been uploaded so if we open up default hyphen Merlin we can see it's using sse-kms as the service side encryption type and it's using the KMS key that we set in the default encryption settings on the bucket.

Okay well that's everything I wanted to cover in this demo lesson so let's just tidy up to make sure that we don't experience any charges go back to the Amazon S3 console select the bucket that you've created and then click on empty you need to confirm to empty that bucket once that process is completed and the bucket's emptied and then follow that same process but this time click on delete to delete the bucket from your AWS account click on key management service and we'll just mark the key that were created for deletion so select customer managed keys select cat pics click on key actions schedule key deletion set this to 7 which is the minimum check the box and click on schedule deletion with that being said though that's everything I wanted to cover in this demonstration I hope it's been fun and useful go ahead mark this video is complete and when you're ready I'll see you in the next video.

Welcome back and in this lesson I want to talk about S3 encryption.

Now we're going to be focusing on server-side encryption known as SSE, which I will be coaching on client-side encryption and how that's different.

Now we've got a lot to get through so let's jump in and get started.

Now before we start there's one common misconception which I want to fix right away, and that's that buckets aren't encrypted, objects are.

You don't define encryption at the bucket level.

There's something called bucket default encryption, but that's different and I'll cover that elsewhere in the course.

For now, understand that you define encryption at the object level, and each object in a bucket might be using different encryption settings.

Now before we talk about the ways that S3 natively handles encryption for objects, I think it's useful to just review the two main architectures of encryption which can be used with the product.

There's client-side encryption and server-side encryption, and both of these refer to what method is used for encryption at rest, and this controls how objects are encrypted as they're written to disk.

It's a method of ensuring that even if somebody were to get the physical disks from AWS which your data is on, they would need something else, a type of key to access that data.

So visually this is how a transaction between a group of users or an application and S3 looks like.

The users of the application on the left are loading data to an S3 endpoint for a specific bucket which gets stored on S3's base storage hardware.

Now it's a simplistic overview, but for this lesson it's enough.

I want to illustrate the difference between client-side encryption and server-side encryption.

So on the top we have client-side encryption, and on the bottom we have server-side encryption.

Now this is a really, really important point which often confuses students.

What I'm talking about in this lesson is encryption at rest, so how data is stored on disk in an encrypted way.

Both of these methods also use encryption in transit between the user-side and S3.

So this is an encrypted tunnel which means that you can't see the raw data inside the tunnel.

It's encrypted.

So ignoring any S3 encryption, ignoring how data is encrypted as it's written to disk, data transferred to S3 and from S3 is generally encrypted in transit.

Now there are exceptions, but use this as your default and I'll cover those exceptions elsewhere in the course.

So in this lesson when we're talking about S3 encryption, we're focusing on encryption at rest and not encryption in transit, which happens anyway.

Now the difference between client-side encryption and server-side encryption is pretty simple to understand when you see it visually.

With client-side encryption, the objects being uploaded are encrypted by the client before they ever leave, and this means that the data is ciphertexted the entire time.

From AWS's perspective, the data is received in a scrambled form and then stored in a scrambled form.

AWS would have no opportunity to see the data in its plain text form.

With server-side encryption known as SSE, it's slightly different.

Here, even though the data is encrypted in transit using HTTPS, the objects themselves aren't initially encrypted, meaning that inside the tunnel, the data is in its original form.

Let's assume it's animal images.

So you could remove the HTTP encrypted tunnel somehow and the animal pictures would be in plain text.

Now once the data hits S3, then it's encrypted by the S3 servers, which is why it's referred to as server-side encryption.

So to high level, the differences are with client-side encryption, everything is yours to control.

You take on all of the risks and you control everything, which is both good and bad.

You take the original data, you are the only one who ever sees the plain text version of that data, you generate a key, you hold that key and you manage that key.

You are responsible for recording which key is used for which object, and you perform the encryption process before it's uploaded to S3, and this consumes CPU capacity on whatever device is performing the encryption.

You just use S3 for storage, nothing else.

It isn't involved in the encryption process in any way, so you own and control the keys, the process and any tooling.

So if your organization needs all of these, if you have real reasons that AWS cannot be involved in the process, then you need to use client-side encryption.

Now with server-side encryption known as SSE, you allow S3 to handle some or all of that process, and this means there are parts that you need to trust S3 with.

How much of that process you trust S3 with and how you want the process to occur and determine which type of server-side encryption you use as there are multiple types.

Now AWS has recently made server-side encryption mandatory, and so you can no longer store objects in an unencrypted form on S3.

You have to use encryption at rest.

So let's break apart server-side encryption and review the differences between each of the various types.

There are three types of server-side encryption available for S3 objects, and each is a trade-off of the usual things, trust, overhead, cost, resource consumption and more.

So let's quickly step through them and look at how they work.

The first is SSE-C, and this is server-side encryption with customer-provided keys.

Now don't confuse this with client-side encryption because it's very different.

The second is SSE-S3, which is server-side encryption with Amazon S3 managed keys, and this is the default.

The last one is an enhancement on SSE-S3, which is SSE-KMS, and this is server-side encryption with KMS keys stored inside the AWS Key Management Service, known as KMS.

Now the difference between all of these methods is what parts of the process you trust S3 with and how the encryption process and key management is handled.

At a high level, there are two components to server-side encryption.

First, the encryption and decryption process.

This is the process where you take plain text, a key and an algorithm, and generate cyber text.

It's also the reverse, so taking that cyber text and a key and using an algorithm to output plain text.

Now this is symmetrical encryption, so the same key is used for both encryption and decryption.

The second component is the generation and management of the cryptographic keys, which are used as part of the encryption and decryption processes.

These three methods of server-side encryption, they handle these two components differently.

Now let's look at how.

Now before we do, again, I just want to stress that SSE is now mandatory on objects within S3 buckets.

This process will occur, you cannot choose not to use it.

The only thing that you can influence is how the process happens and what version of SSE is utilized.

Now first, with SSE-C, the customer is responsible for the keys, and S3 manages the encryption and decryption processes.

So the major change between client-side encryption and this is that S3 are handling the cryptographic operations.

Now this might sound like a small thing, but if you're dealing with millions of objects and a high number of transactions, then the CPU capability required to do encryption can really add up.

So you're essentially offloading the CPU requirements of this process to AWS, but you still need to generate and manage the key or keys.

So when you put an object into S3 using this method, you provide the plain text object and an encryption key.

Remember this object is encrypted in transit by HTTPS on its way to S3, so even though it's plain text right now, it's not visible to an external observer.

When it arrives at S3, the object is encrypted and a hash of the key is tagged to the object and the key is destroyed.

Now this hash is one way, it can't be used to generate a new key, but if a key is provided during decryption, the hash can identify if that specific key was used or not.

So the object and this one-way hash are stored on disk, assistantly.

Remember S3 doesn't have the key at this stage.

To decrypt, you need to provide S3 with the request and the key used to encrypt the object.

If it's correct, S3 decrypts the object, discards the key and returns the plain text.

And again, returning the object is done over an encrypted HTTPS tunnel, so from the perspective of an observer, it's not visible.

Now this method is interesting.

You still have to manage your keys, which does come with a cost and some effort, but you also retain control of that process, which is good in some regulation-heavy environments.

You also save on CPU requirements versus client-side encryption, because S3 performs encryption and decryption, meaning smaller devices don't need to consume resources for this process.

But you need to trust that S3 will discard the keys after use, and there are some independent audits which prove what AWS does and doesn't do during this process.

So you choose SSE-C when you absolutely need to manage your own keys, but are happy to allow S3 to perform the encryption and decryption processes.

You would choose client-side encryption when you need to manage the keys and also the encryption and decryption processes, and you might do this if you never want AWS to have the ability to see your plain text data.

So let's move on to the next type of server-side encryption, and the type I want to describe now is SSE-S3.

And with this method, AWS handles both the encryption processes as well as the key generation and management.

When putting an object into S3, you just provide the plain text data.

When an object is uploaded to S3 using SSE-S3, it's encrypted by a key which is unique for every object, so S3 generates a key just for that object, and then it uses that key to encrypt that object.

For extra safety, S3 has a key which it manages as part of the service.

You don't get to influence this, you can't change any options on this key, nor do you get to pick it.

It's handled end-to-end by S3.

From your perspective, it isn't visible anywhere in the user interface, and it's rotated internally by S3 out of your visibility and control.

This key is used to encrypt the per-object key, and then the original key is discarded.

What we're left with is a ciphertext object and a ciphertext key, and both of these are persistently stored on disk.

With this method, AWS take over the encryption process just as with SSE-C, but they also manage the keys on your behalf, which means even less admin overhead.

The flip side with this method is that you have very little control over the keys used.

The S3 key is outside of your control, and the keys used to encrypt and encrypt objects are also outside of your control.

For most situations, SSE-S3 is a good default type of encryption which makes sense.

It uses a strong algorithm, AES256, the data is encrypted at rest and the customer doesn't have any admin overhead to worry about, but it does present three major problems.

Firstly, if you're in an environment which is strongly regulated, where you need to control the keys used and control access to the keys, then this isn't suitable.

If you need to control rotation of keys, this isn't suitable.

And then lastly, if you need role separation, this isn't suitable.

What I mean by role separation is that a full S3 administrator, somebody who has full S3 permissions to configure the bucket and manage the objects, then he or she can also decrypt and view data.

You can't stop an S3 full administrator from viewing data when using this type of server-side encryption.

And in certain industry areas such as financial and medical, you might not be allowed to have this small and open access for service administrators.

You might have certain groups within the business who can access the data but can't manage permissions, and you might have requirements for another SIS admin group who need to manage the infrastructure but can't be allowed to access data within objects.

And with SSE-S3, this cannot be accomplished in a rigorous best practice way.

And this is where the final type of server-side encryption comes in handy.

The third type of server-side encryption is server-side encryption with AWS Key Management Service Keys, known as SSE-KMS.

How this differs is that we're now involving an additional service, the Key Management Service, or KMS.

Instead of S3 managing keys, this is now done via KMS.

Specifically, S3 and KMS work together.

You create a KMS key, or you can use the service default one, but the real power and flexibility comes from creating a customer-managed KMS key.

It means this is created by you within KMS, it's managed by you, and it has isolated permissions, and I'll explain why this matters in a second.

In addition, the key is fully configurable.

Now this seems on the surface like a small change, but it's actually really significant in terms of the capabilities which it provides.

When S3 wants to encrypt an object using SSE-KMS, it has to liaise with KMS and request a new data encryption key to be generated using the chosen KMS key.

KMS delivers two versions of the same data encryption key, a plain text version and an encrypted or cipher text version.

S3 then takes the plain text object and the plain text data encryption key and creates an encrypted or cipher text object, and then it immediately discards the plain text key, leaving only the cipher text version of that key and both of these are stored on S3 storage.

So you're using the same overarching architecture, the per object encryption key, and the key which encrypts the per object key, but with this type of server-side encryption, so using SSE-KMS, KMS is generating the keys.

Now KMS keys can only encrypt objects up to 4KB in size, so the KMS key is used to generate data encryption keys which don't have those limitations.

It's important to understand that KMS doesn't store the data encryption keys, it only generates them and gives them to S3.

But you do have control over the KMS key, the same control as you would with any other customer-managed KMS key.

So in regulated industries, this alone is enough reason to consider SSE-KMS because it gives fine-grained control over the KMS key being used as well as its rotation.

You also have logging and auditing on the KMS key itself, and with CloudTrail you'll be able to see any calls made against that key.

But probably the best benefit provided by SSE-KMS is the role separation.

To decrypt an object encrypted using SSE-KMS, you need access to the KMS key which was originally used.

That KMS key is used to decrypt the encrypted copy of the data encryption key for that object which is stored along with that object.

If you don't have access to KMS, you can't decrypt the data encryption key, so you can't decrypt the object, and so it follows that you can't access the object.

Now what this means is that if we had an S3 administrator, and let's call him Phil, because we're using SSE-KMS, it means Phil as an S3 administrator does have full control over this bucket.

But because Phil has been given no permissions on the specific KMS key, he can't read any objects.

So he can administer the object as part of administering S3, but he can't see the data within those objects because he can't decrypt the data encryption key using the KMS key because he has no permissions on that KMS key.

Now this is an example of role separation, something which is allowed using SSE-KMS versus not allowed using SSE-S3.

With SSE-S3, Phil as an S3 administrator could administer and access the data inside objects.

However, using SSE-KMS, we have the option to allow Phil to view data in objects or not, something which is controllable by granting permissions or not on specific KMS keys.

So time for a quick summary before we finish this lesson, and it's really important that you understand these differences for any of the AWS exams.

With client-side encryption, you handle the key management and the encryption and decryption processes.

Use this if you need to control both of those and don't trust AWS and their regular audits.

This method uses more resources to manage keys as well as resources for actually performing the encryption and decryption processes at scale.

But it means AWS never see your objects in plain text form because you handle everything end to end.

This generally means you either encrypt all objects in advance or use one of the client-side encryption SDKs within your application.

Now please don't confuse client-side encryption with server-side encryption, specifically SSE-C.

Client-side encryption isn't really anything to do with S3, it's not a form of S3 encryption, it's different.

You can use client-side encryption and server-side encryption together, there's nothing preventing that.

So now let's step through server-side encryption, and remember this is now on by default, it's mandatory.

The only choice you have is which method of SSE to use.

With SSE-C you manage the encryption keys, you can use the same key for everything, but that isn't recommended.

Or you can use individual keys for every single object, the choice is yours.

S3 accepts your choice of key and an object and it handles the encryption and decryption processes on your behalf.

This means you need to trust S3 with the initial plain text object and trust it to discard and not store the encryption key.

But in exchange S3 takes over the computationally heavy encryption and decryption processes.

And also keep in mind that the data is transferred in a form where it's encrypted in transit using HTTBS.

So nobody outside AWS will ever have exposure for plain text data in any way.

SSE-S3 uses AES-256, I mention this because it's often the way exam questions test your knowledge.

If you see AES-256, think SSE-S3.

With SSE-S3, S3 handles the encryption keys and the encryption process.

It's the default and it works well for most cases, but you have no real control over keys, permissions or rotation.

And it also can't handle role separation, meaning S3 for admins can access the data within objects that they manage.

Finally we have SSE-KMS which uses KMS and KMS keys which the service provides.

You can control key rotation and permissions, it's similar in operation to SSE-S3, but it does allow role separation.

So use this if your business has fairly rigid groups of people and compartmentalised sets of security.

You can have S3 admins with no access to the data within objects.

Now for all AWS exams make sure you understand the difference between client side and server side encryption.

And then for server side encryption try and pitch scenarios where you would use each of the three types of server side encryption.

Now that's everything I wanted to cover in this lesson about object encryption, specifically server side encryption.

Go ahead and complete this lesson, but when you're ready I look forward to you joining me in the next.

Welcome back and in this demo lesson I just wanted to give you a bit of practical exposure to KMS.

And in this demo we're going to continue using the scenario that we've got the cat ruler trying to send encrypted battle plans to the robot general.

Now to get started just make sure that you're logged in into the IAM admin user of the general AWS account, so the management account of the organisation.

As always you'll need to have the Northern Virginia region selected and once you do go ahead and move across to the KMS console.

So click in the search box at the top, type KMS and then open that in a new tab.

And then click on create key because we'll be creating a KMS key.

Now KMS allows the creation of symmetric or asymmetric keys and just to keep this demonstration simple we're going to demonstrate using a symmetric key.

So make sure this option selected and then just to demonstrate some of these options just expand this and this is where you can set the key material origin.

If you recall from the previous lesson I talked about how the physical backing key material could be generated by KMS or imported and this is where you select between those two options.

Now I'm not going to talk about the custom key store at this point in the course.

I'll talk about this in more detail later on in the course when I talk about cloud HSM which is a different product entirely.

For now just select KMS and this will make KMS generate this physical backing key material that we'll be using to do our cryptographic operations.

Now historically KMS was a single region service which means that keys created in the product could never leave the region that they were created in.

More recently KMS has extended this functionality allowing you to create multi region keys.

Now for the purpose of this demo lesson we're only going to be using a single region key which is the default.

So make sure that single region key is selected and then we can continue to go ahead and click on next.

Now in the previous lesson I mentioned how a key has a unique ID but also that we could create an alias which references the key.

And that's what we can do here so I'm going to go ahead and create an alias and I'm going to call the alias cat robot all onward.

So type in cat robot and click next.

Now I discussed earlier how a KMS key has a key policy and a key policy is a resource policy that applies to the key.

Now it's here where we can specify the key administrators for a particular key.

There is a difference between identities that can manage a key and identities that can use a key for cryptographic operations like encrypt or decrypt.

It's this point where we define who can manage this key.

So go ahead and just check the box next to I am admin.

So that will make sure that our I am admin user can administer this key.

Once you do that scroll down there's a box here that allows administrators also to delete this key and that's a default so we'll leave that as is and click next.

Now the previous set is where we defined who had admin permissions on the key.

This stage lets us define who can use the key for cryptographic operations so encrypt and decrypt.

To keep things simple we're going to define a key which adds the relevant entries to the key policy.

So just check the box next to I am admin which is the user that we logged in as and then just scroll down.

So if you wanted to add other AWS accounts here so that they had permission to use this key but for this demonstration we don't need to do that.

So just click on next and this is the key policy that this wizard has created.

So if I just scroll down it assigns the account level trust at the top so the account itself, the account user is allowed to perform any KMS column actions on this key.

So that is the part of the policy of this statement here which means that this key will trust this account.

It's this statement that defines the key administrators so that I am admin inside this account can create and describe and enable and list and all the other admin style permissions.

Scroll down further still it's this statement that allows I am admin to perform encryption and decrypt and re-encrypt and generate data key and describe key actions against this key.

So all the permissions that we define are inside the key policy.

So this point go ahead click on finish and that will create the key as well as the alias that we use to reference this key.

If we go into this key I'll just show you some of the options that are available.

We'll be able to obviously edit the key policy and we can define key rotation.

So by default key rotation for a customer managed key is switched off and we can enable it to rotate the key once every year.

For an AWS managed key that is by default turned on and you can't switch it off and it also performs a rotation approximately once every year.

So just click on AWS managed keys as we go through the course and start turning on encryption for various different services.

You'll notice how each service the first time it uses encryption with KMS it creates an AWS managed key in this list.

Now that's everything we need to do on the AWS side.

Now we can start using this key to perform some cryptographic operations.

So let's do that.

Now at this point rather than using the local command line interface on your local machine we're going to be using Cloud Shell.

This allows us to use the same set of instructions regardless of your local operating system.

So to launch Cloud Shell click on this icon and this will take a few minutes but it will put you at the shell that's using your currently logged in user in order to gain permission.

So any commands you run in the shell will be run as your currently logged in user.

So the first thing that we'll be doing is to create a plain text battle plan.

So this is the message that the cab ruler is going to be sending to the robot general.

To generate that file we'll use echo and then space and then a speechmark and then a small message and the message is going to be find all the doggos and then a comma distract them with the yums.

So find all the doggos, distract them with the yums and then a speechmark to close that off and then we'll redirect that to a file called battleplans.txt.

And then presenter.

Now the commands to interact with KMS from the command line are fairly long so what I'll do is paste it in and then I'll step through it line by line and explain exactly what it's doing.

So first we need to encrypt the plain text battle plans and we want the result to be a cipher text document something that we can pass to the robot general which can't be intercepted en route and can be decrypted at the other end.

So this is the command that we need to run and I just want to step through this line by line.

The top part should be fairly obvious so we're running the AWS command line tools, the KMS module and using the encrypt action.

So this specifies that we want to encrypt piece of data.

This line specifies the alias that we want to use to encrypt this piece of data.

You can either specify the key ID which uniquely identifies a key or you can specify an alias using the alias forward slash and then the name of the alias.

In this case I've lept it to do that so this is using the alias that we created in the first part of this demo lesson.

This line is where we're providing the plain text to KMS and instead of typing the plain text on the command line we're telling it to consult this file so battleplans.txt.

Now the next line is telling the command line tools to output the result as text and it's going to be a text output with a number of different fields.

The next line, double hyphen query, is telling the command line tools to select one particular field and that's the field cipher text blob and it's this field that contains the cipher text output from the KMS command.

Now the output of any of these commands that interact with KMS is going to be a base64 encoded file so it's not going to be binary data, it's going to be base64 encoded.

What we want to do is have our output being a binary encrypted file and so we need to take the result of this encryption command and pass it to a utility called base64 and that utility using this command line option will decode that base64 and place the result into a not_battleplans.enc file and this is going to be our result in cipher text.

Now I know that command is relatively complex, KMS is not the easiest part of AWS to use from the command line but I did want to step you through line by line so you didn't know what each line achieved.

Ok so let's go ahead and run this command, to do that we need to click on paste and then once that's pasted into a cloud shell press enter to run the command and the output not_battleplans will be our encrypted cipher text.

So if I run a cat not_battleplans we get binary encrypted data so obviously anyone looking from the outside will just see scrambled data and won't understand what the message is.

So now I'm going to clear the screen to make it a little bit easier to see and this is the encrypted cipher text file that we could transfer across to the robot general.

So now we need to assume in this scenario that we're now the robot general and we're looking to decrypt this file.

Ok so now I'm going to paste the next command for this lesson which is the decrypt command and I'll be stepping through line by line just explaining exactly what each line accomplishes.

So this is the command that you use to decrypt the cipher text and give us the original plain text battle plans.

So first this top line should be logical we're running the AWS command line tools with the KMS module and the decrypt command.

We're passing in some cipher text so we use the command line option double hyphen cipher text blob and instead of pasting this on the command line we're giving it this file so not_battleplans.enc.

We're again asking for the output to be in text.

This will output some text with a number of different fields we're using the query field query for the plain text field and again the output will be base 64 encoded and so we're using the base 64 utility with the double hyphen decode to decode that back into its original form and store that into a file called decryptorplans.txt.

So let's go ahead and run this so click paste and then press enter to run this command.

This will decrypt cipher text and it will output decryptorplans.txt.

And if we catch that document we'll see the original message.

Find all the doggos, distract them with the yums and that's just been a really simple demonstration of using the KMS encrypt command and the KMS decrypt command.

A couple of things I wanted to draw your attention to throughout the process.

With the encrypt command we needed to pass in the key to use as well as the plain text and we got out the cipher text.

With the decrypt command we don't need to specify the key, we only give the cipher text and assuming we have permissions on the KMS key so that we can use it to perform decrypt operations then we'll get the decryptor plain text and that's what's happened here.

Now just to clear up from this lesson if you go back to the AWS console make sure you're in US East 1 so Northern Virginia and go back to the key management service console and we're just going to delete the KMS key that we created earlier in this lesson.

So click on customer managed keys, select the KMS key that we created earlier, my case, cap robot then click on key actions and schedule key deletion.

You need to enter a waiting period between 7 and 30 days since you want this cleared up as fast as possible going into 7, tick the box to confirm and then schedule delete.

And that'll put the key into a pending deletion state and after 7 days it'll be entirely removed.

And at that point we've cleared up all of the assets that we've used in this demo lesson so go ahead and complete the video and when you're ready join me in the next.

Welcome to this video where I'm going to be talking about the key management service known as KMS.

Now this product is used by many other services within AWS when they use encryption, so don't be surprised if you're watching this video in what seems like a pretty random place within the course.

With that being said let's jump in and get started.

Now KMS isn't all that complex as a product.

Once you understand it it's pretty simple, but because of how much it's used by other AWS products and services it's essential that you do understand it for all the AWS exams.

Now KMS is a regional and public service.

Every region is isolated when using KMS.

Think of it as a separate product.

Now KMS is capable of some multi-region features but I'll be covering those in a separate dedicated video.

It's a public service which means it occupies the AWS public zone and can be connected to from anywhere with access to this zone.

Like any other AWS service though you will need permissions to access it.

Now KMS as the name suggests manages keys.

Specifically it lets you create, store and manage cryptographic keys.

These are keys which can be used to convert plain text to ciphertext and vice versa.

Now KMS is capable of handling both symmetric and asymmetric keys.

And at this point you should understand what that means.

Where symmetric keys are used, where public asymmetric are used, as well as private asymmetric.

Just know that KMS is capable of operating with all of these different key architectures.

Now KMS is also capable of performing cryptographic operations which includes, but is not limited to, encryption and encryption operations.

And I'll be talking more about this later in this video.

Now one of the foundational things to understand about KMS is that cryptographic keys never leave the product.

KMS can create keys, keys can be imported, it manages keys, it can use these keys to perform operations but the keys themselves are locked inside the product.

Its primary function is to ensure the keys never leave and held securely within the service.

Now KMS also provides a FIPS 140-2 compliant service.

This is a US security standard but try to memorize this.

It's FIPS 140-2 level 2 to be specific.

Again the level 2 part matters.

It's often a key point of distinction between using KMS versus using something like cloud HSM which I'll be covering in detail elsewhere.

Now some of KMS's features have achieved level 3 compliance but overall it's level 2.

Again please do your best to remember this.

It will come in handy for most of the AWS exams.

Now before we continue, since this is an introduction video, unless I state otherwise, assume that I'm talking about symmetric keys.

When I mention keys within this video, I'm going to be covering the advanced functionality of KMS in other videos including asymmetric keys but for this one I'm mainly focusing on its architecture and high level functions.

So just assume I'm talking about symmetric keys from now on unless I indicate otherwise.

Now the main type of key that KMS manages are known logically enough as KMS keys.

You might see these referred to as CMKs or Customer Master Keys but that naming scheme has been superseded so they're now called KMS keys.

These KMS keys are used by KMS within cryptographic operations.

You can use them, applications can use them and other AWS services can use them.

Now they're logical, think of them as a container for the actual physical key material and this is the data that really makes up the key.

So a KMS key contains a key ID, a creation date, a key policy which is a resource policy, a description and a state of the key.

Every KMS key is backed by physical key material, it's this data which is held by KMS and it's this material which is actually used to encrypt and decrypt things that you give to KMS.

The physical key material can be generated by KMS or imported into KMS and this material contained inside a KMS key can be used to directly encrypt or decrypt data up to 4KB in size.

Now this might sound like a pretty serious limitation.

KMS keys are generally only used to work on small bits of data or to generate other keys and I'll be covering this at a high level later in this video.

Let's look visually at how KMS works so far.

So this is KMS and this is Ashley.

Ashley's first interaction with KMS after picking a region is to create a KMS key.

A KMS key is created with physical backing material and this key is stored within KMS in an encrypted form.

Nothing in KMS is ever stored in plain text form persistently.

It might exist in memory in plain text form but on disk it's encrypted.

Now Ashley's next interaction with KMS might be to request that some data is encrypted.

To do this she makes an encrypt call to KMS specifying the key to use and providing some data to encrypt.

KMS accepts the data and assuming Ashley has permissions to use the key, it decrypts the KMS key then uses this key to encrypt the plain text data that Ashley supplied and then returns that data to Ashley.

Notice how KMS is performing the cryptographic operations.

Ashley is just providing data to KMS together with instructions and it's handling the operations internally.

Logically at some point in the future Ashley will want to decrypt this same data so she calls a decrypt operation and she includes the data she wants to decrypt along with this operation.

KMS doesn't need to be told which KMS key to use for the decrypt operation.

That information is encoded into the cyber text of the data which Ashley wants to decrypt.

The permissions to decrypt are separate from the permissions to encrypt and are also separate from permissions which allow the generation of keys but assuming Ashley has the required permissions for a decrypt operation using this specific KMS key, KMS decrypts the key and uses this to decrypt the cyber text provided by Ashley and returns this data back to Ashley in plain text form.

Now again I want to stress at no point during this entire operation do the KMS keys leave the KMS product.

At no point are the keys stored on the disk in plain text form and at each step Ashley needs permissions to perform the operations and each operation is different.

KMS is very granular with permissions.

You need individual permissions for various operations including encrypt and decrypt and you need permissions on given KMS keys in order to use those keys.

Ashley could have permissions to generate keys and to use keys to encrypt and decrypt or she could have just one of those permissions.

She might have permissions to encrypt data but not decrypt it or she might have permissions to manage KMS creating keys and setting permissions but not permissions to use keys to encrypt or decrypt data and this process is called "Rowl Separation".

Now I mentioned at the start of this lesson that the KMS key can only operate cryptographically on data, which is a maximum of 4kb in size.

Now that's true, so let's look at how KMS gets around this.

Data encryption keys, also known as DEX or D-E-Ks, are another type of key which KMS can generate.

They're generated using a KMS key, using the generate data key operation.

This generates a data encryption key which can be used to encrypt and decrypt data which is more than 4kb in size.

Data encryption keys are linked to the KMS key which created them, so KMS can tell that a specific data encryption key was created using a specific KMS key.

But, and this is pretty much the most important thing about KMS and data encryption keys, KMS doesn't store the data encryption key in any way.

It provides it to you or the service using KMS and then it discards it.

The reason it discards it is that KMS doesn't actually do the encryption or decryption of data using data encryption keys.

You do or the service using KMS performs those operations.

So let's look at how this works.

The data encryption key is generated, KMS provides you with two versions of that data encryption key.

First, a plain text version of that key, something which can be used immediately to perform cryptographic operations.

And second, a ciphertext or encrypted version of that same data encryption key.

The data encryption key is encrypted using the KMS key that generated it.

And in future, this encrypted data encryption key can be given back to KMS for it to be decrypted.

Now the architecture is that you would generate a data encryption key immediately before you wanted to encrypt something.

You would encrypt the data using the plain text version of the data encryption key and then once finished with that process, discard the plain text version of that data encryption key.

That would leave you with the encrypted data and you would then store the encrypted data encryption key along with that encrypted data.

Now a few key things about this architecture.

KMS doesn't actually do the encryption or decryption on data larger than 4KB using data encryption keys.

You do or the service using KMS does.

KMS doesn't track the usage of data encryption keys.

That's also you or the service using KMS.

You could use the same data encryption key to encrypt 100 or a million files or you could request a new data encryption key for each of those million files.

How you decide to do this is based on your exact requirements and of course AWS services will make this choice based on their requirements.

By storing the encrypted data encryption key on disk with the encrypted data, you always have the correct data encryption key to use.

But both the deck and the data are encrypted so administration is easy and security is maintained.

When you're encrypting that data is simple.

You pass the encrypted data encryption key back to KMS and ask for it to decrypt it using the same KMS key used to generate it.

Then you use the decrypted data encryption key that KMS gives you back and decrypt the data with it and then you discard the decrypted data encryption key.

Services such as S3 when using KMS generate a data encryption key for every single object.

They encrypt the object and then discard the plain text version.

As we move through the course I'll be talking in detail about how those services integrate with KMS for encryption services.

Before we finish up with this lesson there are a few key concepts which I want to discuss.

The one thing which is really important to grasp with KMS is that by default KMS keys are stored within the KMS service in that specific region.

They never leave the region and they never leave the KMS service.

You cannot extract a KMS key.

Any interactions with a KMS key are done using the APIs available from KMS.

Now this is the default but KMS does support multi-region keys where keys are replicated to other AWS regions.

But I'll be covering that in a dedicated video if required for the course that you're studying.

In KMS as a product keys are either AWS owned or customer owned.

We're going to be dealing mainly with customer owned keys.

AWS owned keys are a collection of KMS keys that an AWS service owns and manages for use in multiple AWS accounts.

They operate in the background and you largely don't need to worry about them.

If applicable for the course that you're studying I'll have a separate video on this.

If not don't worry it's unimportant.

Now in dealing with customer owned keys there are two types.

AWS managed and customer managed and I'll be covering the specifics of these in a dedicated video.

AWS managed keys are created automatically by AWS when you use a service such as S3 which integrates with KMS.

Customer managed keys are created explicitly by the customer to use directly in an application or within an AWS service.

Customer managed keys are more configurable.

For example you can edit the key policy which means you could allow cross account access so that other AWS accounts can use your keys.

AWS managed keys can't really be customized in this way.

Both types of keys support rotation.

Rotation is where physical backing materials are the data used to actually do cryptographic operations is changed.

With AWS managed keys this can't be disabled.

It's set to rotate approximately once per year.

With customer managed keys rotation is optional.

It's enabled by default and happens approximately once every year.

A KMS key contains the backing key, the physical key material and all previous backing keys caused by rotation.

It means that as a key is rotated data encrypted with all versions can still be decrypted.

Now you can create aliases which are shortcuts to keys.

So you might have an alias called my app one which points at a specific KMS key.

That way KMS keys can be changed if needed.

But be aware the aliases are also per region.

You can create my app one in all regions but in each region it will point at a different key.

Neither aliases or keys are global by default.

Okay to finish up this KMS 101 lesson I want to talk at high level about permissions on KMS keys.

Permissions on keys are controlled in a few ways.

KMS is slightly different than other AWS services that you come across in terms of how keys are handled.

Many services will always trust the account that they're contained in.

Meaning if you grant access via an identity policy that access will be allowed unless there's an explicit deny.

KMS is different.

This account trust is explicitly added on a key policy or not.

The starting point for KMS security is the key policy.

This key policy is a type of resource policy like a bucket policy only on a key.

Every KMS key has one and for custom managed keys you can change it.

To reiterate this the reason the key policy is so important is that unlike other AWS services KMS has to explicitly be told that keys trust the AWS account that they're contained within.

And this is what a key policy might look like.

It means that the key will allow the account 11112222333 to manage it.

This trust isn't automatic so be careful when updating it.

You always need this type of key policy in place if you want to be able to grant access to a key using identity policies.

The key doesn't trust the AWS account and this means that you would need to explicitly add any permissions on the key policy itself.

Generally KMS is managed using this combination of key policies trusting the account and then using identity policies to let IAM users interact with the key.

But in high security environments you might want to remove this account trust and insist on any key permissions being added inside the key policy.

And a typical IAM permissions policy for KMS might look something like this which gives the holder of the policy the rights to use this key to encrypt or decrypt data.

Inside KMS permissions are very granular and can be split based on function.

You can be granted rights to create keys and manage keys but not to have permissions to perform cryptographic operations like encrypt or decrypt.

This way your product administrators are given rights to access data encrypted by KMS which is a common requirement of many higher security environments.

Now there's another way to interact with KMS using grants but I'll be covering this elsewhere in another video if needed.

So that's everything I wanted to cover in this KMS introduction video.

This video is going to form the foundation for others in this series depending on the topic that you're studying there might be no more videos or many more videos.

Don't be worried it'll be the case.

Now at this point that's everything I wanted to talk about though about KMS at this introductory level.

Go ahead and complete the video and when you're ready I look forward to you joining me in the next.

Welcome back.

In this demo lesson, I just want to give you a really quick overview of the S3 performance improvement feature that we talked about in the previous lesson.

Now, I want to make this really brief because it's not something that I can really demonstrate effectively because I have a relatively good internet connection.

So I'm not going to be able to achieve the differences in performance that will effectively demonstrate this feature.

But we can see how to enable it and then use an AWS tool to see exactly what benefits we can expect.

So to enable it, we can move to the S3 console and just create a bucket.

Now, you won't be able to use bucket names with periods in with this accelerator transfer tool.

So this would not be a valid bucket name to use.

It cannot have periods in the name.

So I'm going to go ahead and create a test bucket.

I'm going to call it test AC1279, which is something I don't believe I've used before.

Now, you should go ahead and create a bucket yourself just to see where to enable this, but you can just follow along with what I'm doing.

This won't be required at any point further down the line in the course.

So if you just want to watch me in this demo, that's completely okay.

So I'll create this bucket.

I'll accept the rest of the defaults.

Then I'll select the bucket and go to properties and it's right down at the bottom where you enable transfer acceleration.

So this is an on/off feature.

So if I select it, I can enable transfer acceleration or disable it.

And when I enable it, it provides a new endpoint for this bucket to utilize the transfer acceleration feature.

So that's important.

You need to use this specific endpoint in order to get the benefit of these accelerator transfers.

And this will resolve to an edge location that is highly performant wherever you're physically located in the world.

So it's important to understand that you need to use this endpoint.

So enable this on the bucket and click on save and that's all you need to do.

That enables the feature.

Now, as I mentioned at the start of this lesson, this is not the easiest feature to demonstrate, especially if you have a good internet connection.

This is much better as a demonstration if I was on a suboptimal connection, which I'm not right now.

But what you can do is click on the URL which is attached to this lesson, which opens a comparison tool.

So what this is going to do now is it's going to compare the direct upload speeds that I can achieve from my local machine to this specific S3 bucket.

And it's going to do so with and without transfer acceleration.

So it will be giving you a comparison of exactly what speed you can expect to upload to a specific AWS region, such as US East 1, and then how that compares, uploading to that same region using accelerator transfer.

So we can see already this is demonstrating a 144% faster upload speed from my location to Northern Virginia by using accelerator transfer.

Now, I'm going to allow this to continue running while I continue talking, because what you'll see if you run this tool is a different set of results than what I'm seeing.

You'll see different benefits in each region of using accelerator transfer, depending on your home location.

So if you're located in San Francisco, for example, you probably won't see a great deal of difference between directly uploading and using accelerator transfer.

But for more distant regions, you'll see a much more pronounced improvement.

So if I just move down on this page and make these different regions a little bit easier to see, you'll note, for example, I achieve a much larger benefit in Oregon than I do in San Francisco.

And my result for Dublin, which is even further away from my current location, is a yet higher benefit for using accelerator transfer.

So the less optimal the network route is between your location and the region that's being tested, the better benefit you'll achieve by using accelerator transfer.

Now, there are quite a lot of AWS regions, so I'm not going to let this test finish, but I do recommend if you are interested in S3 performance and this feature specifically, you should test this from your internet connection and allow this process to finish, because it will give you a really good indication of what performance you can expect to each of the AWS regions, and then how S3 accelerator transfer will improve that performance.

Now, that is everything I wanted to cover in this lesson.

I know it's been a brief demo lesson, and it isn't really a demo lesson where you're doing anything practically, but I did just want to supplement the previous lesson by giving you a visual example of how this feature can improve performance to S3.

And I do hope you'll try this tool from your internet connection so you can see the benefit it provides from your location.

With that being said, though, that is everything that I wanted to cover, so go ahead, complete this video, and when you're ready, I'll see you in the next.

Welcome back, and this time we're going to cover a few performance optimization aspects of S3.

If you recall from earlier in the course, this is the Animals For Life scenario.

We have a head office in Brisbane, remote offices which consume services from the Brisbane office, and remote workers using potentially slower or less reliable services to access and upload data to and from the head office.

So keep the scenario in mind as we step through some of the features that S3 offers to improve performance.

It's not always about performance.

It's often about performance and reliability combined.

And this is especially relevant when we're talking about a distributed organization such as Animals For Life.

So let's go ahead and review the features that S3 offers, which help us in this regard.

Now, understanding the performance characteristics of S3 is essential as a solutions architect.

We know from the Animals For Life scenario that remote workers need to upload large data sets and do so frequently.

And we know that they're often on unreliable internet connections.

Now, this is a concern because of the default way that S3 uploads occur.

By default, when you upload an object to S3, it's uploaded as a single blob of data in a single stream.

A file becomes an object, and it's uploaded using the put object API call and placed in a bucket.

And this all happens as a single stream.

Now, this method has its problems.

While it is simple, it means that if a stream fails, the whole upload fails, and the only recovery from it is a full restart of the entire upload.

If the upload fails at 4.5 GB of a 5 GB upload, that's 4.5 GB of data wasted and probably a significant amount of time.

Remember, the data sets are being uploaded by remote workers over slow and potentially unreliable internet links.

And this data is critical to the running of the organization.

Any delay can be costly and potentially risky to animal welfare.

When using this single put method, the speed and reliability of the upload will always be limited because of this single stream of data.

If you've ever downloaded anything online, it's often already using multiple streams behind the scenes.

There are many network-related reasons why even on a fast connection, one stream of data might not be optimal, especially if the transfer is occurring over long distances.

In this type of situation, single stream transfers can often provide much slower speeds than both ends of that transfer are capable of.

If I transfer you data with a single stream, it will often run much slower than my connection can do and your connection can do.

Remember, when transferring data between two points, you're only ever going to experience the speed, which is the lowest of those two points, but often using single stream transfer, you don't even achieve that.

Data transfer protocols such as BitTorrent have been developed in part to allow speedy distributed transfer of data.

And these have been designed to address this very concern.

Using data transfer with only a single stream is just a bad idea.

Now, there is a limit within AWS if you utilize a single put upload, then you're limited to 5 GB of data as a maximum.

But I would never trust a single put upload with anywhere near that amount of data.

It's simply unreliable.

But there is a solution to this.

And that solution is multi-part upload.

Multi-part upload improves the speed and reliability of uploads to S3.

And it does this by breaking data up into individual parts.

So we start with the original blob of data that we want to upload to S3, and we break this blob up into individual parts.

Now, there is a minimum.

The minimum size for using multi-part upload is 100 MB.

So the minimum size for this original blob of data is 100 megabytes.

You can't use multi-part upload if you're uploading data smaller than this.

Now, my recommendation is that you start using this feature the second that you can.

The most AWS tooling will automatically use it as soon as it becomes available, which is at this 100 MB lower threshold.

There are almost no situations where a single put upload is worth it when you get above 100 MB.

The benefits of multi-part upload are just too extensive and valuable.

Now, an upload can be split into a maximum of 10,000 parts.

And each part can range in size between 5 MB and 5 GB.

The last part is left over, so it can be smaller than 5 MB if needed.

Now, the reason why multi-part upload is so effective is that each individual part is treated as its own isolated upload.

So each individual part can fail in isolation and be restarted in isolation, rather than needing to restart the whole thing.

So this means that the risk of uploading large amounts of data to S3 is significantly reduced.

But not only that, it means that because we're uploading lots of different individual bits of data, it improves the transfer rate.

The transfer rate of the whole upload is the sum of all of the individual parts.

So you get much better transfer rates by splitting this original blob of data into smaller individual parts and then uploading them in parallel.

It means that if you do have any single stream limitations on your ISP or any network inefficiencies by uploading multiple different streams of data, then you more effectively use the internet bandwidth between you and the S3 endpoint.

Now, next, I want to talk about a feature of S3 called Accelerated Transfer.

To understand Accelerated Transfer, it's first required to understand how global transfer works to S3 buckets.

Let's use an example.

Let's say that the Animals for Life Organization has a European campaign which is running from the London office.

For this campaign, there'll be data from staff in the field.

And let's say that we have three teams dedicated to this campaign, one in Australia, one in South America, and one on the West Coast of the US.

Now, the S3 bucket, which is being used by the campaign staff, has been created in the London region.

So this is how this architecture locks.

We've got three geographically spread teams who are going to be uploading data to an S3 bucket that's located within the UK.

Now, it might feel like when you upload data to S3, your data would go in a relatively straight line, the most efficient line to its destination.

Now, this is not how networking works.

How networking works is that it is possible for the data to take a relatively indirect path.

And the data can often slow down as it moves from hop to hop on the way to its destination.

In some cases, the data might not be routed the way you expect.

I've had data, for instance, routed from Australia to the UK, but taking the alternative path around the world.

It's often not as efficient as you expect.

Remember, S3 is a public service, and it's also regional.

In the case of the Australian team, their data would have to transit across the public internet all the way from Australia to the UK before it enters the AWS public zone to communicate with S3.

And we have no control over the public internet data path.

Routers and ISPs are picking this path based on what they think is best and potentially commercially viable.

And that doesn't always align with what offers the best performance.

So using the public internet for data transit is never an optimal way to get data from source to destination.

Luckily, as Solutions Architects, we have a solution to this, which is S3 transfer acceleration.

Transfer acceleration uses the network of AWS edge locations, which are located in lots of convenient locations globally.

An S3 bucket needs to be enabled for transfer acceleration.

The default is that it's switched off, and there are some restrictions for enabling it.

The bucket name cannot contain periods, and it needs to be DNS compatible in its naming.

So keep in mind those two restrictions.

But assuming that's the case, once enabled, data being uploaded by our field workers, instead of going back to the S3 bucket directly, it immediately enters the closest, best performing AWS edge location.

Now this part does occur over the public internet, but geographically, it's really close, and it transits through fewer normal networks, so it performs really well.

At this point, the edge locations transit the data being uploaded over the AWS global network, a network which is directly under the control of AWS, and this tends to be a direct link between these edge locations and other areas of the AWS global network, in this case, the S3 bucket.

Remember, the internet is a global, multi-purpose network, so it has to have lots of connections to other networks, and many stops along the way, where traffic is routed from network to network, and this just slows performance down.

Think of the internet as the normal public transit network, when you need to transit from bus to train to bus to bus, to get to a far-flung destination.

The normal transit network, whilst it's not the highest performance, is incredibly flexible, because it allows you to get from almost anywhere to almost anywhere.

The internet is very much like that.

It's not designed primarily for speed.

It's designed for flexibility and resilience.

The AWS network, though, is purpose-built to link regions to other regions in the AWS network, and so this is much more like an express train, stopping at only the source and destination.

It's much faster and with lower consistent latency.

Now, the results of this, in this context, are more reliable and higher performing transfers between our field workers and the S3 bucket.

The improvements can vary, but the benefits achieved by using transfer acceleration improve the larger the distance between the upload location and the location of the S3 bucket.

So in this particular case, transferring data from Australia to a bucket located in Europe, you'll probably see some significant gains by using transfer acceleration.

The worse the initial connection, the better the benefit by using transfer acceleration.

Okay, so now it's time for a demonstration.

In the next lesson, I just want to take a few moments to show you an example of how this works.

I want to show you how to enable the feature on an S3 bucket, and then demonstrate some of the performance benefits that you can expect by using an AWS-provided tool.

So go ahead, finish this video, and when you're ready, you can join me in the demo lesson.

Welcome back and in this lesson I'm going to cover object versioning and MFA delete, two essential features of S3.

These are two things I can almost guarantee will feature on the exam and almost every major project I can involved in has needed solid knowledge of both.

So let's jump in and get started.

Object versioning is something which is controlled at a bucket level.

It starts off in a disabled state.

You can optionally enable versioning on a disabled bucket, but once enabled you cannot disable it again.

Just to be super clear, you can never switch bucket versioning back to disabled once it's been enabled.

What you can do though is suspend it and if desired a suspended bucket can be re-enabled.

It's really important for the exam to remember these stage changes.

So make a point of noting them down and when revising try to repeat until it sticks.

So a bucket starts off as disabled, it can be re-enabled again, an enabled bucket can be moved to suspended and then moved back to enabled.

But the important one is that enabled bucket can never be switched back to disabled.

That is critical to understand for the exam.

So you can see many trick questions which will test your knowledge on that point.

Without versioning enabled on a bucket, each object is identified solely by the object key, its name, which is unique inside the bucket.

If you modify an object, the original version of that object is replaced.

Versioning lets you store multiple versions of an object within a bucket.

Any operations which would modify an object, generate a new version of that object and leave the original one in place.

For example, let's say I have a bucket and inside the bucket is a picture of one of my cats, Winky.

So the object is called Winky.JPEG.

It's identified in the bucket by the key, essentially its name, and the key is unique.

If I modify the Winky.JPEG object or delete it, those changes impact this object.

Now there's an attribute of an object which I haven't introduced yet and that's the ID of the object.

When versioning on a bucket is disabled, the ID of the object in that bucket are set to null.

That's what versioning being off on a bucket means.

All of the objects have an ID of null.

Now if you upload or put a new object into a bucket with versioning enabled, then S3 allocates an ID to that object.

In this case, 111, 111.

If any modifications are made to this object, so let's say somebody accidentally overrides the Winky.JPEG object with the dog picture, but still calls it Winky.JPEG.

S3 doesn't remove the original object.

It allocates a new ID to the newer version and it retains the old version.

The newest version of any object in a version-enabled bucket is known as the current version of that object.

So in this case, the object called Winky.JPEG with an ID of 2222222, which is actually a dog picture, that is the current version of this object.

Now if an object is accessed without explicitly indicating to S3 which version is required, then it's always the current version which will be returned.

But you've always got the ability of requesting an object from S3 and providing the ID of a specific version to get that particular version back rather than the current version.

So versions can be individually accessed by specifying the ID, and if you don't specify the ID, then it's assumed that you want to interact with the current version, the most recent version.

Now versioning also impacts deletions.

Let's say we've got these two different versions of Winky.JPEG stored in a version-enabled bucket.

If we indicate to S3 that we want to delete the object and we don't give any specific version ID, then what S3 will do is try a new special version of that object known as a delete marker.

Now the delete marker essentially is just a new version of that object, so S3 doesn't actually delete anything, but the delete marker makes it look deleted.

In reality though, it's just hidden.

The delete marker is a special version of an object which hides all previous versions of that object.

But you can delete the delete marker which essentially undeletes the object, returning the current version to being active again, and all the previous versions of the object still exist, accessible using their unique version ID.

Now even with versioning enabled, you can actually fully delete a version of an object, and that actually really deletes it.

To do that, you just need to delete an object and specify the particular version ID that you want to remove.

And if you are deleting a particular version of an object and the version that you're deleting is the most recent version, so the current version, then the next most recent version of that object then becomes the current version.

Now some really important points that you need to be aware about object versioning.

I've mentioned this at the start of the lesson, it cannot be switched off, it can only be suspended.

Now why that matters is that when versioning is enabled on a bucket, all the versions of that object stay in that bucket, and so you're consuming space for all of the different versions of an object.

If you have one single object that's 5 gig in size, and you have five versions of that object, then that's 5 times 5 gig of space that you're consuming for that one single object, and it's multiple versions.

And logically, you'll build for all of those versions of all of those objects inside an S3 bucket, and the only way that you can zero those costs out is to delete the bucket and then re-upload all those objects to a bucket without versioning enabled.

That's why it's important that you can't disable versioning.

You can only suspend it, and when you suspend it, it doesn't actually remove any of those old versions, so you're still built for them.

Now there's one other relevant feature of S3 which does make it to the exam all the time, and that's known as MFA delete.

Now MFA delete is something that's enabled within the versioning configuration on a bucket.

And when you enable MFA delete, it means that MFA is required to change bucket versioning state.

So if you move from enable to suspend it or vice versa, you need this MFA to be able to do that, and also MFA is required to delete any versions of an object.

So to fully delete any versions, you need this MFA token.

Now the way that this works is that when you're performing API calls in order to change a bucket to versioning state or delete a particular version of an object, you need to provide the serial number of your MFA token as well as the code that it generates.

You concatenate both of those together, and you pass that along with any API calls to interact how you delete versions or change the versioning state of a bucket.

Okay, so that's all of the theory for object versioning inside S3.

And at this point, that's everything I wanted to cover in this license.

I'll go ahead and complete the video, and when you're ready, I'll look forward to you joining me in the next.

Welcome back in this demo lesson you're going to gain some practical experience of working with the versioning feature of S3.

So to get started just make sure that you're logged in to the management account of the organization, so the general account, and then make sure that you've got the Northern Virginia region selected, so US-EAS-1.

Now there is a link attached to this lesson which you need to click on and then extract.

This is going to contain all of the files that you'll be using throughout this demo.

So go ahead and click on that link, extract it, and it should create a folder called S3_Versioning.

Once you've confirmed that you're logged in and have the right region selected, then go ahead and move to the S3 console.

So you can get there either using the recently visited services, or you can type S3 into the Find Services box and click to move to the S3 console.

Now to demonstrate versioning we're going to go ahead and create an S3 bucket, we're going to set it up for static website hosting, enable versioning, and then experiment with some objects and just observe how versioning changes the default behavior inside an S3 bucket.

So go ahead and click on Create bucket.

As long as the bucket name is unique, its specific name isn't important because we won't be using it with Route 53.

So just give the bucket a name and make sure that it's something unique.

So I'm going to use AC_Bucket_13337.

You should pick something different than me and different from something that any other student would use.

Once you've selected a unique bucket name, just scroll down and uncheck Block All Public Access.

We're going to be using this as a static website hosting bucket, so this is fine.

And we'll need to acknowledge that we understand the changes that we're making, so check this box, scroll down a little bit more, and then under bucket versioning we're going to click to enable versioning.

Keep scrolling down and at the bottom click on Create bucket.

Next, go inside the bucket, click on Properties, scroll all the way down to the bottom, and we need to enable static website hosting.

So click on Edit, check the box to enable static website hosting.

For hosting type, we'll set it to host a static website, and then for the index document, just type index.html, and then for the error document, type error.html.

Once you've set both of those, you can scroll down to the bottom and click on Save Changes.

Now as you learned in the previous demo lesson, just enabling static website hosting isn't enough to allow access, we need to apply a bucket policy.

So click on the permissions tab, scroll down, and under bucket policy click on Edit.

Now inside the link attached to this lesson, which you should have downloaded and extracted, there should be a file called bucket_policy.json, which is an example bucket policy.

So go ahead and open that file and copy the contents into your clipboard, move back to the console and paste it into the policy box, and we need to replace this example bucket placeholder with the ARN for this bucket.

So copy the bucket ARN into your clipboard by clicking this icon.

Because this ARN references objects in this bucket, and we know this because it's got forward slash star at the end, we need to replace only the first part of this placeholder ARN with the actual bucket ARN from the top.

So select from the A all the way up to the T, so not including the forward slash and the star, and then paste in the bucket ARN that you copied onto your clipboard.

Once you've done that, you can scroll down and then click on Save Changes.

Next, click on the objects tab, and we're going to upload some of the files that you downloaded from the link attached to this lesson.

So click on Upload, and first we're going to add the files.

So click on Add Files, then you'll need to go to the location where you downloaded and extracted the file that's attached to this lesson.

And once you're there, go into the folder called S3_Versioning, and you'll see a folder called Website.

Open that folder, select index.html and click on Open, and then click on Add Folder, and select the IMG folder that's also in that same location.

So select that folder and then click on Upload.

So this is going to upload an index.html object, and it's going to upload a folder called IMG which contains winky.jpeg.

Once you've done that, scroll down to the bottom and just click on Upload.

Now once the upload's completed, you can go ahead and click on Close, and what you'll see in the Objects dialog inside the bucket is index.html and then a folder called IMG.

And as we know by now, S3 doesn't actually have folders it uses prefixes, but if we go inside there, you'll see a single object called winky.jpeg.

Now go back to the bucket, and what we're going to do is click on Properties, scroll down to the bottom, and then click on this icon to open our bucket in a new browser tab.

All being well, you should see AnimalsForLife.org, Animal of the Week, and a picture of my one-eyed cat called winky.

So this is using the same architecture as the previous demo lesson where you experienced static website hosting.

What we're going to do now though is experiment with versions.

So go back to the main S3 console, scroll to the top, and click on Objects.

So because we've got versioning enabled on this bucket, as I talked about in the previous theory lesson, it means that every time you upload an object to this S3 bucket, it's assigned a version ID.

And if you upload an object with the same name, then instead of overwriting that object, it just creates a new version of that object.

Now with versioning enabled and using the default settings, we don't see all the individual versions, but we can elect to see them by toggling this Show Versions toggle.

So go ahead and do that.

Now you'll see that every object inside the S3 bucket, you'll see a particular version ID, and this is a unique code which represents this particular version of this particular object.

So if we go inside the IMG folder, you'll see that we have the same for winkeep.jpeg.

Toggle Show Versions to Disable, and you'll see that that version ID disappears.

What I want you to do now is to click on the Upload button inside this IMG folder.

So click on Upload, and then click on Add Files.

Now inside this Lessons folder, so S3 versioning, at the top level you've got a number of folders.

You have Website, which is what you uploaded to this S3 bucket, and this Image folder contains winkeep.jpeg.

So this is a particular file, winkeep.jpeg, that contains the picture of winkeep my one-eyed cat.

Now if you expand version 1 and version 2, you might be able to tell that version 1 is the same one-eyed cat, and we can expand that and say that it is actually winkeep.

Inside version 2 we have an object with the same name, but if we expand this, this is not winkeep, this is a picture of truffles.

So let's say that an administrator of this bucket makes a mistake and uploads this second version of winkeep.jpeg, which is not actually winkeep, it's actually truffles the cat.

But let's say that we do this, so we select winkeep.jpeg from the version 2 folder, and we click on Open.

Once we've selected that for upload, we scroll all the way down to the bottom and click on Upload.

That might take a few seconds to complete the upload because these are relatively large image files, but once it's uploaded you can click on Close.

So now we're still inside this image folder, and if we refresh, all we can see is one object, winkeep.jpeg.

So it looks with this default configuration of the user interface, like we've overwritten a previous object with this new object.

And if we go back to the tab which has got the static website open and hit refresh, you'll see that this image has indeed been replaced by the truffles image.

So even though it's called winkeep.jpeg, this is clearly truffles.

Now if we go back to the S3 console, and now if we enable the versions toggle, now we can see that we've got two different versions of this same object.

We've got the original version at the bottom and a new version at the top.

And note how both of these have different version IDs.

Now what S3 does is it always picks the latest version whenever you use any operations which simply request that one object.

So if we just request the object like we're doing with the static website hosting, then it will always pick the current or the latest version of this object.

But we do still have access to the older versions because we have versioning enabled on this bucket.

Nothing is ever truly deleted as long as we're operating with objects.

So let's experiment with exactly what functionality this gives us.

Go ahead and toggle show versions.

Once you've done that, select the winkeep.jpeg object and then click delete.

You'll need to type or copy and paste delete into this delete objects box and then click on delete.

Before we do that, note what it says at the top.

Deleting the specified objects adds delete markers to them.

If you need to undo the delete action, you can delete the delete markers.

So let's explore what this means.

Go ahead and click on delete objects.

And once it's completed, click on close.

Now how this looks at the moment, we're still in the image folder.

And because we've got show version set to off, it looks like we deleted the object.

But this is not what's occurred because we've got versioning enabled.

What's actually occurred is this is added a new version of this object.

But instead of an actual new version of the object, it's simply added a delete marker as that new version.

So if we toggle show versions back to on, now what we see are the previous versions of winkeep.jpeg.

So the original version at the bottom and the one that we replaced in the middle.

And then at the top we have this delete marker.

Now the delete marker is the thing which makes it look to be deleted in the console UI when we have show version set to off.

So this is how S3 handles deletions when versioning is enabled.

If you're interacting with an object and you delete that object, it doesn't actually delete the object.

It simply adds a delete marker as the most recent version of that object.

Now if we just select that delete marker and then click on delete, that has the effect of undeleting the object.

Now it's important to highlight that because we're dealing with object versions, anything that we do is permanent.

If you're operating with an object and you have versioning enabled on a bucket, if you overwrite it or delete it, all it's going to do is either add a new version or add a delete marker.

When you're operating with versions, everything is permanent.

So in this case we're going to be permanently deleting the delete marker.

So you need to confirm that by the typing or copying and pasting permanently delete into this box and click on delete objects.

What this is going to do is delete the delete marker.

So if we click on close, now we're left with these two versions of winkeep.jpeg so we've deleted the delete marker.

If we toggle show versions to off, we can see that we now have our object back in the bucket.

If we go back to static website hosting and refresh, we can see though that it's still truffle.

So this is a mistake.

It's not actually winky in this particular image.

So what we can do is go back to the S3 console, we can enable show versions.

We know that the most recent version is actually truffles rather than winky.

So what we can do is select this incorrect version, so the most recent version and select delete.

Now again, we're working with an object version.

So this is permanent.

You need to make sure that this is what you intend.

In our case it is.

So you need to either type or copy and paste permanently delete into the box and click on delete objects.

Now this is going to delete the most recent version of this object.

What happens when you do that is it makes the next most recent version of that object the current or latest version.

So now this is the original version of winky.jpeg, the one that we first uploaded to this bucket.

So this is now the only version of this object.

If we go back to the static website hosting tab and hit refresh, this time it loads the correct version of this image.

So this is actually winky my one-eyed cat.

So this is how you can interact with versioning in an S3 bucket.

Whenever it's enabled, it means that whenever you upload an object to the same name instead of overwriting, it simply creates a new version.

Whenever you delete an object, it simply adds a delete marker.

When you're operating with objects, it's always creating new versions or adding delete markers.

But when you're working with particular versions rather than objects, any operations are permanent.

So you can actually delete specific versions of an object permanently and you can delete delete markers to undelete that object.

Now it's not possible to turn off versioning on a bucket.

Once it's enabled on that bucket, you don't have the ability to disable it.

You only have the ability to suspend it.

Now when you suspend it, it stops new versions being created, but it does nothing about the existing versions.

The only way to remove the additional costs for a version-enabled bucket is either to delete the bucket and then reload the objects to a new bucket, or go through the existing bucket and then manually purge any specific versions of objects which aren't required.

So you need to be careful when you're enabling versioning on a bucket because it can cause additional costs.

If you have a bucket where you're uploading objects over and over again, specifically of their large objects, then if you have versioning enabled, you can incur significantly higher costs than if you have a bucket which doesn't have a versioning enabled.

So that's something you need to keep in mind.

If you enable versioning, you need to manage those versions of those objects inside the bucket.

With that being said, let's tidy up.

So let's go back to the main S3 console, select the bucket, click on Empty, copy and paste or type "Permanently Delete" and click on Empty.

When it's finished, click on Exit, and with the bucket still selected, click on Delete.

Copy and paste or type the name of the bucket and confirm it with the delete bucket.

I want you to build out the accounties back in the same state as it was at the start of this demo lesson.

Now at this point, that's everything that I want you to do in this demo lesson.

You've gained some practical exposure with how to deal with object versions inside an S3 bucket.

At this point, go ahead and complete this video, and when you're ready, I'll afford you joining me in the next lesson.

Welcome back.

And in this demo lesson, you're going to get some experience using the S3 static website hosting feature, which I talked about in the previous lesson.

Now, to get started, just make sure that you're logged in to the management account of the organization, and that you're using the IAM Admin user.

So this just makes sure that you have admin permissions over the general or management account of the organization.

Also, make sure that you have the Northern Virginia region selected, which is US-EAST-1.

Normally with S3, when you're interacting with the product, you're doing so using the AWS console UI or the S3 APIs.

And in this demo lesson, you'll be enabling a feature which allows S3 to essentially operate as a web server.

It allows anybody with a web browser to interact with an S3 bucket, load an index page, and load pictures or other media that are contained within that bucket using standard HTTP.

So that's what we're going to do.

And to get started, we need to move across to the S3 console.

So either use S3 in recently visited services, or you can click on the services dropdown, type S3, and then click it in this list.

Now that we're at the S3 console, we're going to create an S3 bucket.

Now, if you chose to register a domain earlier in the course, like I did, so I registered animalsforlife.io, then we're going to connect this S3 bucket with the custom domain that we registered so we can access it using that domain.

If you chose not to use a domain, then don't worry, you can still do this demo.

What you need to do is to go ahead and click on Create a bucket.

Now for the bucket name, if you are not using a custom domain, then you can enter whatever you want in this bucket name as long as it's unique.

If you did register a custom domain and you want to use this bucket with that domain, then you need to enter a DNS formatted bucket name.

So in my case, I'm going to create a bucket which is called Top 10.

It's going to store the world's best cappages, the Top 10 cappages in the world ever.

And it's going to be part of the animalsforlife.io domain.

And so at the end of this, I'm going to add dot and then animalsforlife.io.

And if you've registered your own custom domain, then obviously you need to add your own domain at the end.

You can't use the same name as me.

Once you've entered that name and just scroll down and uncheck Block All Public Access, this is a safety feature of S3.

But because we're intentionally creating an S3 bucket to be used as a static website, we need to uncheck this box.

Now, unchecking this box means that you will be able to grant public access.

It doesn't mean that public access is granted automatically when you uncheck this box.

They're separate steps.

You will, though, need to acknowledge that you understand the risks of unticking that box.

So check this box just to confirm that you understand.

We'll be carefully configuring the security so you don't have to worry about any of those risks.

And once you've set that, we can leave everything else as default.

So just scroll all the way down to the bottom and click on Create Bucket.

So the bucket's been created, but right now, this only allows access using the S3 APIs or the console UI.

So we need to enable static website hosting.

Now, to do that, we're going to click on the bucket.

Once we have the bucket open, we're going to select the Properties tab.

On the Properties tab, scroll all the way down to the bottom.

And right at the very bottom, we've got static website hosting.

And you need to click on the Edit button next to that.

It's a simple yes or no choice at this point.

So check the box to enable static website hosting.

There are a number of different types of hosting.

You can either just host a static website, which is what we'll choose, or you can redirect requests for an object.

So this allows you to redirect to a different S3 bucket.

We'll be covering this later in the course for now.

Just leave this selected.

So host a static website.

Now, in order to use the static website hosting feature, you'll need to provide S3 with two different documents.

The index document is used as the home or default page for the static website hosting.

So if you don't specify a particular object when you're browsing to the bucket, for example, winky.jpg, if you just browse to the bucket itself, then the index document is used.

And we're going to specify index.html.

So this means that an object called index.html will be loaded if we don't specify one.

Now, the error document is used whenever you have any errors.

So if you specify that you want to retrieve an object from the bucket, which doesn't exist, the error document is used.

And for the error document, we're going to call this error.html.

So these two values always need to be provided when you enable static website hosting.

So now we've provided those, we can scroll down and click on save changes.

Now that that feature's enabled, if we just scroll all the way down to the bottom, you'll see that we have a URL for this bucket.

So go ahead and copy that into your clipboard.

We're going to need this shortly.

So this is the URL that you'll use by default to browse to this bucket.

Now, next, what we need to do is to upload some objects to the bucket, which this static website hosting feature is going to use.

Now, to do that, scroll all the way to the top and just click on objects and then click on upload.

So this is the most recent UI version for S3.

And so you have the ability to add files or add folders.

Now, we're going to use both of these.

We're going to use the add files button to add the index.html and the error.html.

And we're going to use the add folder to add a folder of images.

So first, let's do the add files.

So click on add files.

Now, attached to this video is a link which downloads all of the assets that you'll need for this demo.

So go ahead and click on that link to download the zip file and then extract that zip file to your local machine.

And you'll need to move to the folder that you extracted from the zip file.

It should be called static_website_hosting.

So go to that folder.

And then again, there should be a folder in there called website_files.

So go ahead and click on there to go into that folder.

Now, there are three things inside this folder, index.html, error.html and img.

So we'll start by uploading both of these HTML documents.

So select index.html and error.html and then click on open.

And that will add both of these to this upload table.

Next, click on add folder and then select the img folder and click on upload.

So this has prepared all of these different objects ready to upload to this S3 bucket.

If we scroll down, we'll see that the destination for these uploads is our S3 bucket and your name here will be different as long as it's the same as the name you picked for the bucket, that's fine.

Go all the way to the bottom and then go ahead and click on upload.

And that will upload the index.html, the error.html and then the folder called img as well as the contents of that folder.

So at this point, that's all of the objects uploaded to the S3 bucket and we can go ahead and click on close.

So now let's try browsing to this bucket using static website hosting.

So go ahead and click on properties, scroll all the way down to the bottom and here we've got the URL for this S3 bucket.

So go ahead and copy this into your clipboard, open a new tab and then open this URL or click on this symbol to open it in a new tab.

What you'll see is a 403 forbidden error and this is an access denied.

You're getting this error because you don't have any permissions to access the objects within this S3 bucket.

Remember, S3 is private by default and just because we've enabled static website hosting doesn't mean that we have any permissions to access the objects within this S3 bucket.

We're accessing this bucket as an anonymous or unauthenticated user.

So we have no method of providing any credentials to S3 when we're accessing objects via static website hosting.

So we need to give permissions to any unauthenticated or anonymous users to access the objects within this bucket.

So that's the next thing we need to do.

We need to grant permissions to be able to read these objects to any unauthenticated user.

So how do we do that?

The third method is to use a bucket policy.

So that's what I'm gonna demonstrate in order to grant access to these objects.

Now to add a bucket policy, we need to select the permissions tab.

So click on permissions and then below block public access, there's a box to specify a bucket policy.

So click on edit and we need to add a bucket policy.

Now also in the folder that you extracted from this lessons zip file is a file called bucket_policy.json and this is a generic bucket policy.

So this bucket policy has an effect of allow and it applies to any principle because we have this star wild card and because the effect is allow, it grants any principle, the ability to use the S3 get object action which allows anyone to read an object inside an S3 bucket and it applies to this resource.

So this is a generic template, we need to update it, but go ahead and copy it into your clipboard, go back to the S3 console and paste it into this box.

Now we need to replace this generic ARN, so this example bucket ARN.

So what I want you to do is to copy this bucket ARN at the top of the screen.

So copy this into your clipboard and we need to replace part of this template ARN with what we've just copied.

Now an important point to highlight is that this ARN has forward slash star on the end because this ARN refers to any objects within this S3 bucket.

So we need to select only the part before the forward slash.

So starting at the A and then ending at the end of example bucket and then just paste in the ARN at our bucket that we just copied into our clipboard.

What you should end up with is this full ARN with the name of the bucket that you created and then forward slash star.

And once you've got that, go ahead and click on save changes.

This applies a bucket policy which allows any principle, so even unauthenticated principles, the ability to get any of the objects inside this bucket.

So this means that any principle will be able to read objects inside this bucket.

At this point, assuming everything's okay, if you've still got the tab open to the bucket, then go back to that tab and hit refresh.

And what you should see is the top 10 animals in the world.

So position number one, we've got Merlin.

At position number two, we've got Merlin again.

Position number three, another Merlin.

Four, still Merlin.

And then Merlin again at number five.

At number six, we've got Boris.

So the token non Merlin cat.

Number seven, Samson, another token non Merlin cat.

And then number eight, we've got different cat one.

He looks quite a lot like Merlin.

Number nine, different cat two, again, kind of looks like Merlin.

And then number 10, we've got the family.

And then you might not have guessed this, but this entire top 10 contest was judged by, you guessed it, Merlin.

So what you're loading here is the index.html document inside the bucket.

So we haven't specified an object to load.

And because of that, it's using the index document that we specified on the bucket.

We can load the same object by typing specifically index.html on the end, and that will load in the same object.

Now, if we specify an object which doesn't exist, so let's say we used wrong index.html, then instead of the index document, now it's going to load the error document.

So this is the error document that you specified, which is loading error.html.

So this is just an example of how you can configure an S3 bucket to act as a standard static website.

So what it's doing is loading in the index.html object inside the bucket.

And that index.html is loading in images, which are also stored in the bucket.

So if I right click and copy the image location and open this in a new tab, this is essentially just loading this image from the same S3 bucket.

So it's loading it from this folder called img, and it's called Merlin.jpeg.

It's just an object loading from within the bucket.

Now if I go back to the S3 console and just move across to the properties tab and then scroll down, so far in this lesson, you've been accessing this bucket using the bucket website endpoint.

So this is an endpoint that's derived from the name of the bucket.

Now your URL will be different because you will have called your bucket name something else.

Now if you chose to register a custom domain name at the start of this course, you can customize this further.

As long as you call the bucket the same as the DNS name that you want to use, you can actually use Route 53 to assign a custom DNS name for this bucket.

So this part of the demo you'll only be able to do if you've registered a domain within Route 53.

If you haven't, you can skip to the end of this demo where we're going to tidy up.

But if you want to customize this using Route 53, then you can click on the services dropdown and type Route 53 and then click to move to the Route 53 console.

Once you're there, you can click on hosted zones and you should have a hosted zone that matches the domain that you registered at the start of the course.

Go inside that and click on create record.

Now we're going to be creating a simple routing record.

So make sure that's selected and then click on next.

And we're going to define a simple record.

Now I'm going to type the first part of the name of the bucket.

So I used top10.animalsforlive.io is my bucket name.

So I'm going to put top 10 in this box.

Now, because we want to point this at our S3 bucket, we need to choose an endpoint in this dropdown.

So click in this dropdown and then scroll down and we're going to pick alias to S3 website endpoint.

So select that.

Next, you need to choose the region and you should have created the S3 bucket in the US East 1 region because this is the default for everything that we do in the course.

So go ahead and type US-EAS-1 and then select US East Northern Virginia and you should be able to click in enter S3 endpoint and select your bucket name.

Now, if you don't see your bucket here, then either you've picked the wrong region or you've not used the same name in this part of the record name as you picked for your bucket.

So make sure this entire name, so this component plus the domain that you use matches the name that you've selected for the bucket.

Assuming it does, you should be able to pick your bucket in this dropdown.

Once you've selected it, go ahead and click on define simple record.

And once that's populated in the box, click on create records.

Now, once this record's created, you might have to wait a few moments, but you should find that you can then open this bucket using this full DNS name.

So there we go.

It opens up the same bucket.

So we've used Route 53 and we've integrated it using an alias to our S3 endpoint.

Now, again, you can only do this if you create a bucket with the same name as the fully qualified domain name that we just configured.

So this is an example of a fully qualified domain name.

Now, this is the host component of DNS and this is the domain component.

So together they make up a fully qualified domain name and for this to work, you need to create an S3 bucket with the same bucket name as this fully qualified domain name.

And that's what I did at the start of this lesson, which is why it works for me.

And as long as you've done the same, as long as you've registered a custom domain, as long as you've called the bucket the same as what you're creating within Route 53, then you should be able to reference that bucket and then access it using this custom URL.

At this point, we're going to tidy up.

So go back to the Route 53 console and select this record that you've created and then click on delete.

You'll need to confirm it by clicking delete again.

Then we need to go back to the S3 console, select the bucket that you've created, click on empty, and you'll need to either type or copy and paste, permanently delete into this box, and then click on empty.

It'll take a few minutes to empty the bucket.

Once it's completed, click on exit.

And with the bucket still selected, click on delete to delete the bucket.

And you'll need to confirm that by either typing or copy and pasting the name of the bucket and then click delete bucket.

Now, at this point, that's everything that you need to do in this lesson.

It's just an opportunity to experience the theory that you learned in the previous lesson.

Now, there's a lot more that you can do with static website hosting and I'll be going into many more complex examples later on in the course.

But for now, this is everything that you need to do.

So go ahead and complete this video.

And when you're ready, I'll look forward to you joining me in the next.

Welcome back.

In this lesson, I want to talk about a feature of S3 which I use all the time, personally and when I'm doing consulting work for clients. That is S3 static website hosting.

Until this point, we've been accessing S3 via the normal method which is using the AWS APIs. You might not have realized that, but that's how the AWS CLI tools and the console UI work behind the scenes. For instance, to access any objects within S3, we're using the S3 APIs, and assuming we're authenticated and authorized, we use the get object API call to access those resources.

Now, accessing S3 using APIs is useful in certain situations because it's secure and flexible. But using static website hosting can make S3 infinitely more useful because it allows access via standard HTTP, like individuals using a web browser. So, you can use it to host almost anything, for example, a simple blog.

Using static website hosting is pretty simple. You enable it, and in doing so, you have to set an index document and an error document. When you're using a website, if you access a particular page, say, for example, cats_are_amazing.html, then you will get access specifically to that page. If you don't specify a page, for example, netflix.com, you get what's called an index page, which is a default page returned to you when you aren't asking for anything specific. This is the entry point to most websites.

So, when enabling static website hosting on an S3 bucket, we have to point the index document at a specific object in the S3 bucket. The error document is the same, but it's used when something goes wrong. So, if you access a file which isn't there or there is another type of server-side error, that's when the error document is shown.

Now, both of these need to be HTML documents because the static website hosting feature delivers HTML files. When you enable this feature on a bucket, AWS creates a static website hosting endpoint, and this is a specific address that the bucket can be accessed from using HTTP. The exact name of this endpoint is influenced by the bucket name that you choose and the region that the bucket is in. You don't get to select this name; it's automatically generated by those two things.

Now, you can use your own custom domain for a bucket, but if you do want to do that, then your bucket name matters. You can only use a custom domain with a bucket if the name of the bucket matches the domain. So, if I wanted to have a website called top10.animalsforlife.org, then my bucket name would need to be called top10.animalsforlife.org, and that's why I mentioned at the start of the course to get into the habit of reserving your website names by creating S3 buckets using those names.

Static website hosting is great for things like hosting static websites such as blogs, but it's also good for other things. Let's take a look at two common examples. There are two specific scenarios which are perfect for S3: Offloading and out-of-band pages.

With offloading, let's say you have a website which has a top 10 leaderboard of all of the best animals, and it runs on a computer service. Let's assume this is easy too, for now. The computer service does a few things: it delivers a dynamic HTML page and it delivers static media, in this example, images. That dynamic HTML page might need access to a database, so that's not suitable for static S3 hosting, but the static media that's sitting there waiting to be delivered, and in most cases, it probably makes up over 95% of the data volume that the computer service is delivering and likely almost all of the storage space.

Computer services tend to be relatively expensive, so we can offload a lot of this to S3. What we can do is we can take all of the images, so all of the media that the computer service hosts, and we can move that media to an S3 bucket that uses static website hosting, so one which has static website hosting enabled. Then, when the computer service generates the HTML file and delivers this to the customer's browser, this HTML file points at the media that's hosted on the S3 bucket, so the media is retrieved from S3, not the computer service. S3 is likely to be much cheaper for the storage and delivery of any media versus computer service. S3 is custom-designed for the storage of large data at scale, and so generally, whenever you've got an architecture such as this, you should always consider offloading any large data to S3.

Now, the other benefit that I wanted to specifically highlight is out-of-band pages. Now, out-of-band is an old telecommunications term. In an IT context, it generally means a method of accessing something that is outside of the main way. So, for example, you might use out-of-band server management, and this lets you connect to a management card that's in a server using the cellular network. That way, if the server is having networking issues with the normal access methods of the normal network, then you can still access it.

In the context of this example, top10.animalsforlife.org might refer to an error or a status notification system. For example, with this example, the favorite animals page for Animals for Life. If that service was hosted on a computer service such as EC2, and we wanted a maintenance page to show during scheduled or unscheduled maintenance periods, it wouldn't make much sense to have this on the same server because if the server is being worked on, then it's inherently offline. Additionally, putting it on a different EC2 instance is also risky because if EC2 itself has issues, it might not let us show a status page.

So what we do is we use out-of-band pages in this context and another service. If the server was offline for maintenance or it was experiencing stability or performance bugs, then we could change our DNS and point customers at a backup static website hosted on S3, and this could provide a status message or maybe the details for our business's support team.

Now, the pricing structure for S3, once you understand it, is very simple, but it's formed of a number of major components. First, we've got the cost to store data on S3, and this is generally expressed as a per-gigabyte-month fee. So to store a gigabyte of data on S3 for one month, there's a certain cost, and if you store data for less than one month, then you only pay that component, and if you store less than one gig, you only pay that component, so it's a per-gig-month charge.

Now, there's also a data transfer fee, so for every gigabyte of data that you transfer in and out of S3, there's a cost. Now, to transfer data into S3 is always free, so you're never charged for transferring data into S3. To transfer data out of S3, there is a per-gigabyte charge. Now, for storage and data transfer, they are both incredibly cheap. It's sort of the cheapest storage that's available, especially if you're storing large amounts of data, but there is also a third component that you do need to be aware of, and this is especially important when you're using static website hosting, and this is that you're charged a certain amount for requesting data. So, every time you perform an operation, every time you get, every time you list, every time you put, that's classed as an operation, and different operations in S3 have different costs per 1,000 operations.

Now, the reason I mention this is if you're using static website hosting, you're generally not going to store a lot of data. You're also generally not going to transfer a lot of data because what's stored in the S3 bucket is likely to be very small, but if you have a large customer base and if this out-of-band website or this offloading bucket is actually being used heavily by your system, then you could be using a lot of requests, and so you need to be aware of the request charge per S3.

Now, in terms of what's provided in the free tier, you're given 5 GB of monthly storage inside S3. You're allowed 20,000 GET requests and 2,000 PUT requests, so that will cover us for the demo lesson that we're going to do inside this course and probably most of the other activities that we'll do throughout the course, but if you're going to use S3 for any real-world usage, then you will be billed for that usage.

Now, I run a personal blog, control.io, and that runs from S3 using the static website hosting feature, and because I post certification articles, it does get some fairly heavy use. Now, in the entire time that I've run my personal blog, I think the most that I've ever been charged for S3 usage is 17 cents in one month. So when I talk about being charged for S3, I'm going to mention it whenever we go beyond this free tier, but keep in mind that often I'm talking about really tiny amounts of money relative to the value that you're getting. So, you can store a lot in S3 and use it to deliver a lot of data and often be charged a really tiny amount of money, often something that isn't noticeable on the bill of a production aid of the US account.

Okay, that's enough on the theory of static website hosting, so now it's time for a demo, and in this demo, we're going to be using S3 to create a simple static website. Now, I think this demo is going to be a useful one because it brings together a few of the theory concepts that I've been talking about over the last few lessons. So, go ahead, mark this video as complete, and when you're ready, you can join me in the next demo lesson.

Welcome back and in this lesson I want to start talking about S3 security in more detail. Starting with bucket policies which are a type of AWS resource policy. So by now you know the drill, let's jump in and get started.

Now before we start I want to repeat one thing and you have heard me say this before, but I'm going to say it again over and over. S3 is private by default. Everything that we can do to control S3 permissions is based on this starting point. The only identity which has any initial access to an S3 bucket is the account root user of the account which owns that bucket, so the account which created it. Anything else, so any other permissions have to be explicitly granted. And there are a few ways that this can be done.

The first way is using an S3 bucket policy. And an S3 bucket policy is a type of resource policy. A resource policy is just like an identity policy, but as the name suggests, they're attached to resources instead of identities, in this case an S3 bucket. Resource policies provide a resource perspective on permissions. The difference between resource policies and identity policies is all about this perspective. With identity policies you're controlling what that identity can access. With resource policies you're controlling who can access that resource. So it's from an inverse perspective. One is identities and one is resources.

Now identity policies have one pretty significant limitation. You can only attach identity policies to identities in your own account. And so identity policies can only control security inside your account. With identity policies you have no way of giving an identity in another account access to an S3 bucket. That would require an action inside that other account. Resource policies allow this. They can allow access from the same account or different accounts because the policy is attached to the resource and it can reference any other identities inside that policy. So by attaching the policy to the resource and then having flexibility to be able to reference any other identity, whether they're in the same account or different accounts, resource policies therefore provide a great way of controlling access for a particular resource, no matter what the source of that access is.

Now think about that for a minute because that's a major benefit of resource policies, the ability to grant other accounts access to resources inside your account. They also have another benefit, resource policies can allow or deny anonymous principals. Identity policies by design have to be attached to a valid identity in AWS. You can't have one attached to nothing. Resource policies can be used to open a bucket to the world by referencing all principals, even those not authenticated by AWS. So that's anonymous principals. So bucket policies can be used to grant anonymous access.

So two of the very common uses for bucket policies are to grant access to other AWS accounts and anonymous access to a bucket. Let's take a look at a simple visual example of a bucket policy because I think it will help you understand how everything fits together. There's a demo lesson coming up soon where you'll implement one as part of the mini project. So you will get some experience soon enough of how to use bucket policies.

Let's say that we have an AWS account and inside this account is a bucket called Secret Cat Project. Now I can't say what's inside this bucket because it's a secret, but I'm sure that you can guess. Now attached to this bucket is a bucket policy. Resource policies have one major difference to identity policies and that's the presence of an explicit principal component. The principal part of a resource policy defines which principals are affected by the policy. So the policy is attached to a bucket in this case, but we need a way to say who is impacted by the configuration of that policy. Because a bucket policy can contain multiple statements, there might be one statement which affects your account and one which affects another account, as well as one which affects a specific user, the principal part of a policy or more specifically the principal part of a statement in a policy defines who that statement applies to, which identity is which principals.

Now in an identity policy this generally isn't there because it's implied that the identity which the policy is applied to is the principal. That's logical right? Your identity policy by definition applies to you so you are the principal. So a good way of identifying if a policy is a resource policy or an identity policy is the presence of this principal component. If it's there it's probably a resource policy. In this case the principal is a wild card, a star, which means any principal. So this policy applies to anyone accessing the S3 bucket.

So let's interpret this policy. Well first the effect is allow and the principal is star, so any principal. So this effect allows any principal to perform the action S3 get object on any object inside the secret cat project S3 bucket. So in effect it allows anyone to read any objects inside this bucket. So this would equally apply to identities in the same AWS account as the bucket. It could also apply to other AWS accounts, or partner account. And crucially it also applies to anonymous principals. So principals who haven't authenticated to AWS. Bucket policies should be your default thought when it comes to granting anonymous access to objects in buckets and they're one way of granting external accounts that same access. They can also be used to set the default permissions on a bucket. If you want to grant everyone access to Boris's picture for example and then grant certain identities extra rights or even deny certain rights then you can do that. Bucket policies are really flexible. They can do many other things.

So let's quickly just look at a couple of common examples. Bucket policies can be used to control who can access objects even allowing conditions which block specific IP addresses. In this example this bucket policy denies access to any objects in the secret cat project bucket unless your IP address is 1.3.3.7. The condition block here means this statement only applies if this condition is true. So if your IP address, the source IP address is not 1.3.3.7, then the statement applies and access is denied. If your IP address is 1.3.3.7, then this condition is not met because it's a not IP address condition. So if your IP address is this IP address, the condition is not matched and you get any other access that's applicable. Essentially this statement, which is a deny, does not apply.

Now, bucket policies can be much more complex. In this example, one specific prefix in the bucket, remember this is what a folder really is inside a bucket, so one specific prefix called Boris is protected with MFA. It means that accesses to the Boris folder in the bucket are denied if the identity that you're using does not use MFA. The second statement allows read access to objects in the whole bucket. Because an explicit deny overrides an allow, the top statement applies to just that specific prefix in the bucket, so just Boris. Now, I won't labour on about bucket policies because we'll be using them a fair bit throughout the course, but they can range from simple to complex. I will include a link in the lesson description with some additional examples that you can take a look through if you're interested.

In summary though, a resource policy is associated with a resource. A bucket policy, which is a type of resource policy, is logically associated with a bucket, which is a type of resource. Now, there can only be one bucket policy on a bucket, but it can have multiple statements. If an identity inside one AWS account is accessing a bucket, also in that same account, then the effective access is a combination of all of the applicable identity policies plus the resource policy, so the bucket policy. For any anonymous access, so access by an anonymous principal, then only the bucket policy applies, because logically, if it's an anonymous principal, it's not authenticated and so no identity policies apply.

Now, if an identity in an external AWS account attempts to access a bucket in your account, your bucket policy applies as well as anything that's in their identity policies. So there's a two-step process if you're doing cross-account access. The identity in their account needs to be able to access S3 in general and your bucket, and then your bucket policy needs to allow access from that identity, so from that external account.

Now, there is another form of S3 security. It's used less often these days, but I wanted to cover it anyway. Access control lists or ACLs are ways to apply security to objects or buckets. There is a sub-resource of that object or of that bucket. Remember in the S3 introduction lesson earlier in the course, I talked about sub-resources. Well, this is one of those sub-resources. Now, I almost didn't want to talk about ACLs because they are legacy. AWS don't even recommend their use and prefer that you use bucket policies or identity policies. But as a bare minimum, I want you to be aware of their existence.

Now, part of the reason that they aren't used all that often and that bucket policies have replaced much of what they do is that they're actually inflexible and only allow very simple permissions. They can't have conditions like bucket policies and so you're restricted to some very broad conditions. Let me show you what I mean. This is an example of what permissions can be controlled using an ACL. Now, apologies for the wall of text, but I think it's useful to visualize it all at once. There are five permissions which can be granted in an ACL. Read, write, readACP, writeACP and full control. That's it. So it's already significantly less flexible than an identity or a resource policy. What these five things do depend on if they're applied to a bucket or an object. Read permissions, for example, on a bucket allow you to list all objects in that bucket, whereas write permissions on a bucket allow the grantee, which is the principal being granted those permissions, the ability to overwrite and delete any object in that bucket. Read permissions on an object allow the grantee just to read the object specifically as well as its metadata.

Now, with ACLs you either configure an ACL on the bucket, or you configure the ACL on an object. But you don't have the flexibility of being able to have a single ACL that affects a group of objects. You can't do that. That's one of the reasons that a bucket policy is significantly more flexible. It is honestly so much less flexible than a bucket policy to the extent where I won't waste your time with it anymore. It's legacy, and I suspect at some point it won't be used anymore. If there are any specific places in the course which do require knowledge of ACLs, I'll mention it. Otherwise, it's best to almost ignore the fact that they exist.

Now, before we finish up, one final feature of S3 permissions, and that's the block public access settings. In the overall lifetime of the S3 product, this was actually added fairly recently, and it was added in response to lots of public PR disasters where buckets were being configured incorrectly and being set so that they were open to the world. This resulted in a lot of data leaks, and the root cause was a mixture of genuine mistakes or administrators who didn't fully understand the S3 permissions model.

So consider this example, an S3 bucket with resource permissions granting public access. Until block public access was introduced, if you had public access configured, the public could logically access a bucket. Public access in this sense is read only to any objects defined in a resource policy on a bucket, so there's no restrictions. Public access is public access. Block public access added a further level of security, another boundary. And on this boundary is the block public access settings, which apply no matter what the bucket policies say, but they apply to just the public access, so not any of the defined AWS identities. So these settings will only apply to an anonymous principal, somebody who isn't an AWS identity, attempt to access a bucket using these public access configurations.

Now these settings can be set when you create the bucket and adjusted afterwards. They're pretty simple to understand. You can choose the top option which blocks any public access to the bucket, no matter what the resource policy says. It's a full override, a failsafe. Or you can choose the second option which allows any public access granted by any existing ACLs when you enable the setting but it blocks any new ones. The third option blocks any public access granted by ACLs no matter if it was enabled before or after the block public access settings were enabled. The fourth setting allows any existing public access granted by bucket policies or access point policies so anything enabled at the time when you enable this specific block public access setting, they're allowed to continue but it blocks any new ones. The fifth option blocks both existing and new bucket policies from granting any public access.

Now they're simple enough and they function as a final failsafe. If you're ever in a situation where you've granted some public access and it doesn't work, these are probably the settings which are causing that inconsistency. And don't worry, I'll show you where these are accessed in the demo lesson.

Now before we finish up, just one final thing I want to cover and this is an exam at PowerUp. So these are just some key points on how to remember all of the theory that I've discussed in this lesson. When I first started in AWS, I found it hard to know from instinct when to use identity policies versus resource policies versus ACLs. Choosing between resource policies and identity policies much of the time is a preference thing. So do you want to control permissions from the perspective of a bucket or do you want to grant or deny access from the perspective of the identities accessing a bucket? Are you looking to configure one user accessing 10 different buckets or 100 users accessing the same bucket? It's often a personal choice. A choice on what makes sense for your situation and business. So there's often no right answer but there are some situations where one makes sense over the other.

If you're granting or denying permissions on lots of different resources across an AWS account, then you need to use identity policies because not every service supports resource policies. And besides, you would need a resource policy for each service so that doesn't make sense if you're controlling lots of different resources. If you have a preference for managing permissions all in one place, that single place needs to be IAM, so identity policies would make sense. IAM is the only single place in AWS you can control permissions for everything. You can sometimes use resource policies but you can use IAM policies all the time. If you're only working with permissions within the same account so no external access, then identity policies within IAM are fine because with IAM you can only manage permissions for identities that you control in your account. So there are a wide range of situations where IAM makes sense and that's why most permissions control is done within IAM. But there are some situations which are different. You can use bucket policies or resource policies in general if you're managing permissions on a specific product. So in this case S3. If you want to grant a single permission to everybody accessing one resource or everybody in one account, then it's much more efficient to use resource policies to control that base level permission. If you want to directly allow anonymous identities or external identities from other AWS accounts to access a resource, then you should use resource policies.

Now finally, and I know this might seem like I'm anti-access control list, which is true, but so are AWS, never use ACLs unless you really need to. And even then, consider if you can use something else. At this point in time, if you are using an ACL, you have to be pretty certain that you can't use anything else because they're legacy and their inflexible and AWS are actively recommending against their use. So keep that in mind.

Okay, well that's all of the theory that I wanted to cover in this lesson. I know it's been a lot, but we do have to cover this detailed level of security because it's needed in the exam. And you'll be using it constantly throughout the rest of this section and the wider course. At this point, though, go ahead and complete this video. And when you're ready, you can join me in the next where I'm going to be talking about another exciting feature of S3.

Welcome to this lesson, where I'm going to very briefly talk about a special type of IAM role, and that's service-linked roles.

Now, luckily there isn't a great deal of difference between service-linked roles and IAM roles. They're just used in a very specific set of situations. So let's jump in and get started.

So simply put, a service-linked role is an IAM role linked to a specific AWS service. They provide a set of permissions which is predefined by a service. These permissions allow a single AWS service to interact with other AWS services on your behalf.

Now, service-linked roles might be created by the service itself, or the service might allow you to create the role during the setup process of that service. Service-linked roles might also get created within IAM.

The key difference between service-linked roles and normal roles is that you can't delete a service-linked role until it's no longer required. This means it must no longer be used within that AWS service. So that's the one key difference.

In terms of permissions needed to create a service-linked role, here's an example of a policy that allows you to create a service-linked role.

You'll notice a few key elements in this policy. The top statement is an allow statement. The action is iam:CreateServiceLinkedRole. For the resource, it has SERVICE-NAME.amazonaws.com.

The important thing here is not to try to guess this, as different services express this in different ways. The formatting can differ, and it's case-sensitive. I've included a link with an overview of these details attached to this lesson.

When creating this type of policy to allow someone to create service-linked roles, you have to be careful to ensure you do not guess this element of a statement.

Another important consideration with service-linked roles is role separation. When I talk about role separation, I'm not using it in a technical sense, but in a job role sense.

Role separation is where you might give one group of people the ability to create roles and another group the ability to use them. For instance, we might want to give Bob, one of our users, the ability to use a service-linked role with an AWS service.

This involves using the architecture of being able to take a service-linked role and assign it to a service. If you want to give Bob the ability to use a preexisting role with a service but not create or edit that role, you would need to provide Bob with PassRole permissions. This allows Bob to pass an existing role into an AWS service. It's an example of role separation, meaning Bob could configure a service with a role that has already been created by a member of the security team. Bob would just need ListRole and PassRole permissions on that specific role.

This is similar to when you use a pre-created role, for example, with a CloudFormation stack. By default, when creating a CloudFormation stack, CloudFormation uses the permissions of your identity to interact with AWS. This means you need permissions not only to create a stack but also to create the resources that the stack creates. However, you can give, for example, a user like Bob the ability to pass a role into CloudFormation. That role could have permissions that exceed those which Bob directly has. So a role that Bob uses could have the ability to create AWS resources that Bob does not. Bob might have access to create a stack and pass in a role, but the role provides CloudFormation with the permissions needed to interact with AWS.

PassRole is a method inside AWS that allows you to implement role separation, and it's something you can also use with service-linked roles. This is something I wanted to reiterate to emphasize that passing a role is a very important AWS security architecture.

That is everything I wanted to cover in this very brief lesson. It's really just an extension of what you've already learned about IAM roles, and it's something you'll use in demo lessons elsewhere in the course.

For now, I just want you to be aware of how service-linked roles differ from normal roles and how the PassRole architecture works. With that being said, that's everything I wanted to cover in this video.

So go ahead and complete the video, and when you're ready, I look forward to you joining me in the next.

Welcome back, and in this video, I want to talk about AWS Control Tower. This is a product which is becoming required knowledge if you need to use AWS in the real world. And because of this, it's starting to feature more and more in all of the AWS exams. I want this to be a lesson applicable to all of the AWS study paths, so think of this as a foundational lesson. And if required, for the course that you're studying, I might be going into additional detail. We do have a lot to cover, so let's jump in and get started.

At a high level, Control Tower has a simple but wide-ranging job, and that's to allow the quick and easy setup of multi-account environments. You might be asking, "Doesn't AWS Organizations already do that?" Well, kind of. Control Tower actually orchestrates other AWS services to provide the functionality that it does, and one of those services is AWS Organizations. But it goes beyond that. Control Tower uses Organizations, IAM Identity Center, which is the product formerly known as AWS SSO. It also uses CloudFormation, AWS Config, and much more. You can think of Control Tower as another evolution of AWS Organizations adding significantly more features, intelligence, and automation.

There are a few different parts of Control Tower which you need to understand, and it's worth really focusing on understanding the distinction now because we're going to be building on this later. First, we've got the Landing Zone, and simply put, this is the multi-account environment part of Control Tower. This is what most people will be interacting with when they think of Control Tower. Think of this like AWS Organizations only with superpowers. It provides, via other AWS services, single sign-on and ID Federation so you can use a single login across all of your AWS accounts, and even share this with your existing corporate identity store. And this is provided using the IAM Identity Center, again, the service formerly known as AWS SSO. It also provides centralized logging and auditing, and this uses a combination of CloudWatch, CloudTrail, AWS Config, and SNS. Everything else in the Control Tower product surrounds this Landing Zone, and I'll show you how this looks later in this lesson.

Control Tower also provides guardrails, again, more detail on this is coming up soon. But these are designed to either detect or mandate rules and standards across all AWS accounts within the Landing Zone. You also have the Account Factory, which provides really cool automation for account creation, and adds features to standardize the creation of those accounts. This goes well beyond what AWS Organizations can do on its own, and I'll show you how this works over the rest of this lesson. And if applicable, for the path that you're studying, there will be a demo coming up elsewhere in the course. Finally, there's a dashboard which offers a single-page oversight of the entire organization. At a high level, that's what you get with Control Tower.

Now, things always make more sense visually, so let's step through this high-level architecture visually, and I hope this will add a little bit more context. We start with Control Tower itself, which like AWS Organizations, is something you create from within an AWS account. And this account becomes the management account at the Landing Zone. At this top, most level within the management account, we have Control Tower itself, which orchestrates everything. We have AWS Organizations, and as you've already experienced, this provides the multi-account structure, so organizational units and service control policies. And then, we have single sign-on provided by the IAM Identity Center, which historically was known as AWS SSO. This allows for, as the name suggests, single sign-on, which means we can use the same set of internal or federated identities to access everything in the Landing Zone that we have permissions to. This works in much the same way as AWS SSO worked, but it's all set up and orchestrated by Control Tower.

When Control Tower is first set up, it generally creates two organizational units. The foundational organizational units, which by default is called Security, and a custom organizational unit, which by default is named Sandbox. Inside the foundational or security organizational unit, Control Tower creates two AWS accounts, the Audit account and the Log Archive account. The Log Archive account is for users that need access to all logging information for all of your enrolled accounts within the Landing Zone. Examples of things used within this account are AWS Config and CloudTrail logs, so they're stored within this account so that they're isolated. You have to explicitly grant access to this account, and it offers a secure, read-only Archive account for logging.

The Audit account is for your users who need access to the audit information made available by Control Tower. You can also use this account as a location for any third-party tools to perform auditing of your environment. It's in this account that you might use SNS for notifications of changes to governance and security policies, and CloudWatch for monitoring Landing Zone wide metrics. It's at this point where Control Tower becomes really awesome because we have the concept of an Account Factory. Think of this as a team of robots who are creating, modifying, or deleting AWS accounts as your business needs them. And this can be interacted with both from the Control Tower console or via the Service Catalog.

Within the custom organizational unit, Account Factory will create AWS accounts in a fully automated way as many of them as you need. The configuration of these accounts is handled by Account Factory. So, from an account and networking perspective, you have baseline or cookie-cutter configurations applied, and this ensures a consistent configuration across all AWS accounts within your Landing Zone. Control Tower utilizes CloudFormation under the covers to implement much of this automation, so expect to see stacks created by the product within your environment. And Control Tower uses both AWS Config and Service Control Policies to implement account guardrails. And these detect drifts away from governance standards, or prevent those drifts from occurring in the first place.

At a high level, this is how Control Tower looks. Now the product can scale from simple to super complex. This is a product which you need to use in order to really understand. And depending on the course that you're studying, you might have the opportunity to get some hands-on later in the course. If not, don't worry, that means that you only need this high-level understanding for the exam.

Let's move on and look at the various parts of Control Tower in a little bit more detail. Let's quickly step through the main points at the Landing Zone. It's a feature designed to allow anyone to implement a well-architected, multi-account environment, and it has the concept of a home region, which is the region that you initially deploy the product into, for example, us-east-1. You can explicitly allow or deny the usage of other AWS regions, but the home region, the one that you deploy into, is always available. The Landing Zone is built using AWS Organizations, AWS Config, CloudFormation, and much more. Essentially, Control Tower is a product which brings the features of lots of different AWS products together and orchestrates them.

I've mentioned that there's a concept of the foundational OU, by default called the Security OU, and within this, Log Archive and Audit AWS accounts. And these are used mainly for security and auditing purposes. You've also got the Sandbox OU which is generally used for testing and less rigid security situations. You can create other organizational units and accounts, and for a real-world deployment of Control Tower, you're generally going to have lots of different organizational units. Potentially, even nested ones to implement a structure which works for your organization.

Landing Zone utilizes the IAM Identity Center, again, formerly known as AWS SSO, to provide SSO or single sign-on services across multiple AWS accounts within the Landing Zone, and it's also capable of ID Federation. And ID Federation simply means that you can use your existing identity stores to access all of these different AWS accounts. The Landing Zone provides monitoring and notifications using CloudWatch and SNS, and you can also allow end users to provision new AWS accounts within the Landing Zone using Service Catalog.

This is the Landing Zone at a high level. Let's next talk about guardrails. Guardrails are essentially rules for multi-account governance. Guardrails come in three different types: mandatory, strongly recommended, or elective. Mandatory ones are always applied. Strongly recommended are obviously strongly recommended by AWS. And elective ones can be used to implement fairly niche requirements, and these are completely optional.

Guardrails themselves function in two different ways. We have preventative, and these stop you doing things within your AWS accounts in your Landing Zone, and these are implemented using Service Control policies, which are part of the AWS Organizations product. These guardrails are either enforced or not enabled, so you can either enforce them or not. And if they're enforced, it simply means that any actions defined by that guardrail are prevented from occurring within any of your AWS accounts. An example of this might be to allow or deny usage of AWS regions, or to disallow bucket policy changes within accounts inside your Landing Zone.

The second functional type of guardrail is detective, and you can think of this as a compliance check. This uses AWS Config rules and allows you to check that the configuration of a given thing within an AWS account matches what you define as best practice. These type of guardrails are either clear, in violation, or not enabled. And an example of this would be a detective guardrail to check whether CloudTrail is enabled within an AWS account, or whether any EC2 instances have public IPv4 addresses associated with those instances. The important distinction to understand here is that preventative guardrails will stop things occurring, and detective guardrails will only identify those things. So, guardrails are a really important security and governance construct within the Control Tower product.

Lastly, I want to talk about the Account Factory itself. This is essentially a feature which allows automated account provisioning, and this can be done by either cloud administrators or end users with appropriate permissions. And this automated provisioning includes the application of guardrails, so any guardrails which are defined can be automatically applied to these automatically provisioned AWS accounts.

Because these accounts can be provisioned by end users, think of these as members of your organization, then either these members of your organization or anyone that you define can be given admin permissions on an AWS account which is automatically provisioned. This allows you to have a truly self-service, automatic process for provisioning AWS accounts so you can allow any member of your organization within tightly controlled parameters to be able to provision accounts for any purpose which you define as okay. And that person will be given admin rights over that AWS account. These can be long-running accounts or short-term accounts. These accounts are also configured with standard account and network configuration. If you have any organizational policies for how networking or any account settings are configured, these automatically provisioned accounts will come with this configuration. And this includes things like the IP addressing used by VPCs within the accounts, which could be automatically configured to avoid things like addressing overlap. And this is really important when you're provisioning accounts at scale.

The Account Factory allows accounts to be closed or repurposed, and this whole process can be tightly integrated with a business's SDLC or software development life cycle. So, as well as doing this from the console UI, the Control Tower product and Account Factory can be integrated using APIs into any SDLC processes that you have within your organization. If you need accounts to be provisioned as part of a certain stage of application development, or you want accounts to be provisioned as part of maybe client demos or software testing, then you can do this using the Account Factory feature.

At this point, that is everything I wanted to cover at this high level about Control Tower. If you need practical experience of Control Tower for the course that you are studying, there will be a demo lesson coming up elsewhere in the course, which gives you that practical experience. Don't be concerned if this is the only lesson that there is, or if there's this lesson plus additional deep-dive theory. I'll make sure, for whatever course you're studying, you have enough exposure to Control Tower.

With that being said, though, that is the end of this high-level video. So go ahead and complete the video, and when you're ready, I'll look forward to you joining me in the next.

Welcome back and welcome to this CloudTrail demo where we're going to set up an organizational trail and configure it to log data for all accounts in our organization to S3 and CloudWatch logs.

The first step is that you'll need to be logged into the IAM admin user of the management account of the organization. As a reminder, this is the general account. To set up an organizational trail, you always need to be logged into the management account. To set up individual trails, you can do that locally inside each of your accounts, but it's always more efficient to use an organizational trail.

Now, before we start the demonstration, I want to talk briefly about CloudTrail pricing. I'll make sure this link is in the lesson description, but essentially there is a fairly simple pricing structure to CloudTrail that you need to be aware of.

The 90-day history that's enabled by default in every AWS account is free. You don't get charged for that; it comes free by default with every AWS account. Next, you have the ability to get one copy of management events free in every region in each AWS account. This means creating one trail that's configured for management events in each region in each AWS account, and that comes for free. If you create any additional trails, so you get any additional copies of management events, they are charged at two dollars per 100,000 events. That won't apply to us in this demonstration, but you need to be aware of that if you're using this in production.

Logging data events comes at a charge regardless of the number, so we're not going to enable data events for this demo lesson. But if you do enable it, then that comes at a charge of 10 cents per 100,000 events, irrespective of how many trails you have. This charge applies from the first time you're logging any data events.

What we'll be doing in this demo lesson is setting up an organizational trail which will create a trail in every region in every account inside the organization. But because we get one for free in every region in every account, we won't incur any charges for the CloudTrail side of things. We will be charged for any S3 storage that we use. However, S3 also comes with a free tier allocation for storage, which I don't expect us to breach.

With that being said, let's get started and implement this solution. To do that, we need to be logged in to the console UI again in the management account of the organization. Then we need to move to the CloudTrail console. If you've been here recently, it will be in the Recently Visited Services. If not, just type CloudTrail in the Find Services box and then open the CloudTrail console.

Once you're at the console, you might see a screen like this. If you do, then you can just click on the hamburger menu on the left and then go ahead and click on trails. Now, depending on when you're doing this demo, if you see any warnings about a new or old console version, make sure that you select the new version so your console looks like what's on screen now.

Once you're here, we need to create a trail, so go ahead and click on create trail. To create a trail, you're going to be asked for a few important pieces of information, the first of which is the trail name. For trail name, we're going to use "animals4life.org," so just go ahead and enter that. By default, with this new UI version, when you create a trail, it's going to create it in all AWS regions in your account. If you're logged into the management account of the organization, as we are, you also have the ability to enable it for all regions in all accounts of your organization. We're going to do that because this allows us to have one single logging location for all CloudTrail logs in all regions in all of our accounts, so go ahead and check this box.

By default, CloudTrail stores all of its logs in an S3 bucket. When you're creating a trail, you have the ability to either create a new S3 bucket to use or you can use an existing bucket. We're going to go ahead and create a brand new bucket for this trail. Bucket names within S3 need to be globally unique, so it needs to be a unique name across all regions and across all AWS accounts. We're going to call this bucket starting with "CloudTrail," then a hyphen, then "animals-for-life," another hyphen, and then you'll need to put a random number. You’ll need to pick something different from me and different from every other student doing this demo. If you get an error about the bucket name being in use, you just need to change this random number.

You're also able to specify if you want the log files stored in the S3 bucket to be encrypted. This is done using SSE-KMS encryption. This is something that we'll be covering elsewhere in the course, and for production usage, you would definitely want to use it. For this demonstration, to keep things simple, we're not going to encrypt the log files, so go ahead and untick this box.

Under additional options, you're able to select log file validation, which adds an extra layer of security. This means that if any of the log files are tampered with, you have the ability to determine that. This is a really useful feature if you're performing any account-level audits. In most production situations, I do enable this, but you can also elect to have an SNS notification delivery. So, every time log files are delivered into this S3 bucket, you can have a notification. This is useful for production usage or if you need to integrate this with any non-AWS systems, but for this demonstration, we'll leave this one unchecked.

You also have the ability, as well as storing these log files into S3, to store them in CloudWatch logs. This gives you extra functionality because it allows you to perform searches, look at the logs from a historical context inside the CloudWatch logs user interface, as well as define event-driven processes. You can configure CloudWatch logs to scan these CloudTrail logs and, in the event that any particular piece of text occurs in the logs (e.g., any API call, any actions by a user), you can generate an event that can invoke, for example, a Lambda function or spawn some other event-driven processing. Don't worry if you don't understand exactly what this means at this point; I'll be talking about all of this functionality in detail elsewhere in the course. For this demonstration, we are going to enable CloudTrail to put these logs into CloudWatch logs as well, so check this box. You can choose a log group name within CloudWatch logs for these CloudTrail logs. If you want to customize this, you can, but we're going to leave it as the default.

As with everything inside AWS, if a service is acting on our behalf, we need to give it the permissions to interact with other AWS services, and CloudTrail is no exception. We need to give CloudTrail the ability to interact with CloudWatch logs, and we do that using an IAM role. Don’t worry, we’ll be talking about IAM roles in detail elsewhere in the course. For this demonstration, just go ahead and select "new" because we're going to create a new IAM role that will give CloudTrail the ability to enter data into CloudWatch logs.

Now we need to provide a role name, so go ahead and enter "CloudTrail_role_for_CloudWatch_logs" and then an underscore and then "animals_for_life." The name doesn’t really matter, but in production settings, you'll want to make sure that you're able to determine what these roles are for, so we’ll use a standard naming format. If you expand the policy document, you'll be able to see the exact policy document or IAM policy document that will be used to give this role the permissions to interact with CloudWatch logs. Don’t worry if you don’t fully understand policy documents at this point; we’ll be using them throughout the course, and over time you'll become much more comfortable with exactly how they're used. At a high level, this policy document will be attached to this role, and this is what will give CloudTrail the ability to interact with CloudWatch logs.

At this point, just scroll down; that's everything that we need to do, so go ahead and click on "next." Now, you'll need to select what type of events you want this trail to log. You’ve got three different choices. The default is to log only management events, so this logs any events against the account or AWS resources (e.g., starting or stopping an EC2 instance, creating or deleting an EBS volume). You've also got data events, which give you the ability to log any actions against things inside resources. Currently, CloudTrail supports a wide range of services for data event logging. For this demonstration, we won't be setting this up with data events initially because I’ll be covering this elsewhere in the course. So, go back to the top and uncheck data events.

You also have the ability to log insight events, which can identify any unusual activity, errors, or user behavior on your account. This is especially useful from a security perspective. For this demonstration, we won’t be logging any insight events; we’re just going to log management events. For management events, you can further filter down to read or write or both and optionally exclude KMS or RDS data API events. For this demo lesson, we’re just going to leave it as default, so make sure that read and write are checked. Once you've done that, go ahead and click on "next." On this screen, just review everything. If it all looks good, click on "create trail."

Now, if you get an error saying the S3 bucket already exists, you'll just need to choose a new bucket name. Click on "edit" at the top, change the bucket name to something that's globally unique, and then follow that process through again and create the trail.

Certainly! Here is the continuation and completion of the transcript:

---

After a few moments, the trail will be created. It should say "US East Northern Virginia" as the home region. Even though you didn't get the option to select it because it's selected by default, it is a multi-region trail. Finally, it is an organizational trail, which means that this trail is now logging any CloudTrail events from all regions in all accounts in this AWS organization.

Now, this isn't real-time, and when you first enable it, it can take some time for anything to start to appear in either S3 or CloudWatch logs. At this stage, I recommend that you pause the video and wait for 10 to 15 minutes before continuing, because the initial delivery of that first set of log files through to S3 can take some time. So pause the video, wait 10 to 15 minutes, and then you can resume.

Next, right-click the link under the S3 bucket and open that in a new tab. Go to that tab, and you should start to see a folder structure being created inside the S3 bucket. Let's move down through this folder structure, starting with CloudTrail. Go to US East 1 and continue down through this folder structure.

In my case, I have quite a few of these log files that have been delivered already. I'm going to pick one of them, the most recent, and just click on Open. Depending on the browser that you're using, you might have to download and then uncompress this file. Because I'm using Firefox, it can natively open the GZ compressed file and then automatically open the JSON log file inside it.

So this is an example of a CloudTrail event. We're able to see the user identity that actually generates this event. In this case, it's me, I am admin. We can see the account ID that this event is for. We can see the event source, the event name, the region, the source IP address, the user agent (in this case, the console), and all of the relevant information for this particular interaction with the AWS APIs are logged inside this CloudTrail event.

Don’t worry if this doesn’t make a lot of sense at this point. You’ll get plenty of opportunities to interact with this type of logging event as you go through the various theory and practical lessons within the course. For now, I just want to highlight exactly what to expect with CloudTrail logs.

Since we’ve enabled all of this logging information to also go into CloudWatch logs, we can take a look at that as well. So back at the CloudTrail console, if we click on Services and then type CloudWatch, wait for it to pop up, locate Logs underneath CloudWatch, and then open that in a new tab.

Inside CloudWatch, on the left-hand menu, look for Logs, and then Log Groups, and open that. You might need to give this a short while to populate, but once it does, you should see a log group for the CloudTrail that you’ve just created. Go ahead and open that log group.

Inside it, you’ll see a number of log streams. These log streams will start with your unique organizational code, which will be different for you. Then there will be the account number of the account that it represents. Again, these will be different for you. And then there’ll be the region name. Because I’m only interacting with the Northern Virginia region, currently, the only ones that I see are for US East 1.

In this particular account that I’m in, the general account of the organization, if I look at the ARN (Amazon Resource Name) at the top or after US East 1 here, this number is my account number. This is the account number of my general account. So if I look at the log streams, you’ll be able to see that this account (the general account) matches this particular log stream. You’ll be able to do the same thing in your account. If you look for this account ID and then match it with one of the log streams, you'll be able to pull the logs for the general AWS account.

If I go inside this particular log stream, as CloudTrail logs any activity in this account, all of that information will be populated into CloudWatch logs. And that’s what I can see here. If I expand one of these log entries, we’ll see the same formatted CloudTrail event that I just showed you in my text editor. So the only difference when using CloudWatch logs is that the CloudTrail events also get entered into a log stream in a log group within CloudWatch logs. The format looks very similar.

Returning to the CloudTrail console, one last thing I want to highlight: if you expand the menu on the left, whether you enable a particular trail or not, you’ve always got access to the event history. The event history stores a log of all CloudTrail events for the last 90 days for this particular account, even if you don’t have a specific trail enabled. This is standard functionality. What a trail allows you to do is customize exactly what happens to that data. This area of the console, the event history, is always useful if you want to search for a particular event, maybe check who’s logged onto the account recently, or look at exactly what the IAM admin user has been doing within this particular AWS account.

The reason why we created a trail is to persistently store that data in S3 as well as put it into CloudWatch logs, which gives us that extra functionality. With that being said, that’s everything I wanted to cover in this demo lesson.

One thing you need to be aware of is that S3, as a service, provides a certain amount of resource under the free tier available in every new AWS account, so you can store a certain amount of data in S3 free of charge. The problem with CloudTrail, and especially organizational trails, is that they generate quite a large number of requests. There is also, in addition to space, a number of requests per month that are part of the free tier.

If you leave this CloudTrail enabled for the duration of your studies, for the entire month, it is possible that this will go slightly over the free tier allocation for requests within the S3 service. You might see warnings that you’re approaching a billable threshold, and you might even get a couple of cents of bill per month if you leave this enabled all the time. To avoid that, if you just go to Trails, open up the trail that you’ve created, and then click on Stop Logging. You’ll need to confirm that by clicking on Stop Logging, and at that point, no logging will occur into the S3 bucket or into CloudWatch logs, and you won’t experience those charges.

For any production usage, the low cost of this service means that you would normally leave it enabled in all situations. But to keep costs within the free tier for this course, you can, if required, just go ahead and stop the logging. If you don’t mind a few cents per month of S3 charges for CloudTrail, then by all means, go ahead and leave it enabled.

With that being said, that’s everything I wanted to cover in this demo lesson. So go ahead, complete the lesson, and when you're ready, I look forward to you joining me in the next.

Welcome to this lesson, where I'm going to be introducing CloudTrail.

CloudTrail is a product that logs API actions which affect AWS accounts. If you stop an instance, that's logged. If you change a security group, that's logged too. If you create or delete an S3 bucket, that's logged by CloudTrail. Almost everything that can be done to an AWS account is logged by this product.

Now, I want to quickly start with the CloudTrail basics. The product logs API calls or account activities, and every one of those logged activities is called a CloudTrail event. A CloudTrail event is a record of an activity in an AWS account. This activity can be an action taken by a user, a role, or a service.

CloudTrail by default stores the last 90 days of CloudTrail events in the CloudTrail event history. This is an area of CloudTrail which is enabled by default in AWS accounts. It's available at no cost and provides 90 days of history on an AWS account.

If you want to customize CloudTrail in any way beyond this 90-day event history, you need to create a trail. We'll be looking at the architecture of a trail in a few moments' time.

CloudTrail events can be one of three different types: management events, data events, and insight events. If applicable to the course you are studying, I'll be talking about insight events in a separate video. For now, we're going to focus on management events and data events.

Management events provide information about management operations performed on resources in your AWS account. These are also known as control plane operations. Think of things like creating an EC2 instance, terminating an EC2 instance, creating a VPC. These are all control plane operations.

Data events contain information about resource operations performed on or in a resource. Examples of this might be objects being uploaded to S3 or objects being accessed from S3, or when a Lambda function is invoked. By default, CloudTrail only logs management events because data events are often much higher volume. Imagine if every access to an S3 object was logged; it could add up pretty quickly.

A CloudTrail trail is the unit of configuration within the CloudTrail product. It's a way you provide configuration to CloudTrail on how to operate. A trail logs events for the AWS region that it's created in. That's critical to understand. CloudTrail is a regional service.

When you create a trail, it can be configured to operate in one of two ways: as a one-region trail or as an all-regions trail. A single-region trail is only ever in the region that it's created in, and it only logs events for that region. An all-regions trail, on the other hand, can be thought of as a collection of trails in every AWS region, but it's managed as one logical trail. It also has the additional benefit that if AWS adds any new regions, the all-regions trail is automatically updated.

This is a specific configuration item on a trail which determines if it only logs events for the region that it's in or if it also logs global services events. Most services log events in the region where the event occurred. For example, if you create an EC2 instance in AP Southeast 2, it’s logged to that region. A trail would need to be either a one-region trail in that region or an all-regions trail to capture that event.

A very small number of services log events globally to one region. For example, global services such as IAM, STS, or CloudFront are very globally-focused services and always log their events to US East 1, which is Northern Virginia. These types of events are called global service events, and a trail needs to have this enabled in order to log these events. This feature is normally enabled by default if you create a trail inside the user interface.

AWS services are largely split up into regional services and global services. When these different types of services log to CloudTrail, they either log in the region that the event is generated in or they log to US East 1 if they are global services. So, when you're diagnosing problems or architecting solutions, if the logs you are trying to reach are generated by global services like IAM, STS, or CloudFront, these will be classified as global service events and that will need to be enabled on a trail.

Otherwise, a trail will only log events for the isolated region that it’s created in. When you create a trail, it is one of two types: one-region or all-regions. A one-region trail is always isolated to that one region, and you would need to create one-region trails in every region if you wanted to do it manually. Alternatively, you could create an all-regions trail, which encompasses all of the regions in AWS and is automatically updated as AWS adds new regions.

Once you’ve created a trail, management events and data events are all captured by the trail based on whether it's isolated to a region or set to all regions. For an all-region trail, it captures management events and, if enabled, data events. Data events are not generally enabled by default and must be explicitly set when creating a trail. This trail will then listen to everything that's occurring in the account.

Remember that the CloudTrail event history is limited to 90 days. However, when you create a trail, you can be much more flexible. A trail by default can store the events in a definable S3 bucket, and the logs generated and stored in an S3 bucket can be stored there indefinitely. You are only charged for the storage used in S3. These logs are stored as a set of compressed JSON log files, which consume minimal space. Being JSON formatted, they can be read by any tooling capable of reading standard format files, which is a great feature of CloudTrail.

Another option is that CloudTrail can be integrated with CloudWatch Logs, allowing data to be stored in that product. CloudTrail can take all the logging data it generates and, in addition to putting it into S3, it can also put it into CloudWatch Logs. Once it's in CloudWatch Logs, you can use that product to search through it or use a metric filter to take advantage of the data stored there. This makes it much more powerful and gives you access to many more features if you use CloudWatch Logs versus S3.

One of the more recent additions to the CloudTrail product is the ability to create an organizational trail. If you create this trail from the management account of an organization, it can store all the information for all the accounts inside that organization. This provides a single management point for all API and account events across every account in the organization, which is super powerful and makes managing multi-account environments much easier.

So, we need to talk through some important elements of CloudTrail point by point. CloudTrail is enabled by default on AWS accounts, but it’s only the 90-day event history that’s enabled by default. You don’t get any storage in S3 unless you configure a trail. Trails are how you can take the data that CloudTrail’s got access to and store it in better places, such as S3 and CloudWatch Logs.

The default for trails is to store management events only, which includes management plane events like creating an instance, stopping an instance, terminating an instance, creating or deleting S3 buckets, and logins to the console. Anything interacting with AWS products and services from a management perspective is logged by default in CloudTrail. Data events need to be specifically enabled and come at an extra cost. I’ll discuss this in more detail in the demo lesson, as you need to be aware of the pricing of CloudTrail. Much of the service is free, but there are certain elements that do carry a cost, especially if you use it in production.

Most AWS services log data to the same region that the service is in. There are a few specific services, such as IAM, STS, and CloudFront, which are classified as true global services and log their data as global service events to US East 1. A trail needs to be enabled to capture that data.

That’s critical and might come up as an exam question. What you will also definitely find coming up as an exam-style question is where to use CloudTrail for real-time logging. This is one of the limitations of the product—it is not real-time. CloudTrail typically delivers log files within 15 minutes of the account activity occurring and generally publishes log files multiple times per hour. This means it's not real-time; you can't rely on CloudTrail to provide a complete and exhaustive list of events up to the very point you're looking. Sometimes, it takes a few minutes for the data to arrive in S3 or CloudWatch Logs. Keep this in mind if you face any exam questions about real-time logging—CloudTrail is not the product.

Okay, so that's the end of the theory in this lesson. It's time for a demo. In the next lesson, we’ll be setting up an organizational trail within our AWS account structure. We’ll configure it to capture all the data for all our member accounts and our management account, storing this data in an S3 bucket and CloudWatch Logs within the management account. I can’t wait to get started. It’s a fun one and will prove very useful for both the exam and real-world usage.

So go ahead, complete this video, and when you're ready, you can join me in the demo lesson.

Welcome to this lesson, where I'm going to introduce the theory and architecture of CloudWatch Logs.

I've already covered the metrics side of CloudWatch earlier in the course, and I'm covering the logs part now because you'll be using it when we cover CloudTrail. In the CloudTrail demo, we'll be setting up CloudTrail and using CloudWatch Logs as a destination for those logs. So, you'll need to understand it, and we'll be covering the architecture in this lesson. Let's jump in and get started.

CloudWatch Logs is a public service. The endpoint to which applications connect is hosted in the AWS public zone. This means you can use the product within AWS VPCs, from on-premises environments, and even other cloud platforms, assuming that you have network connectivity as well as AWS permissions.

The CloudWatch Logs product allows you to store, monitor, and access logging data. Logging data, at a very basic level, consists of a piece of information, data, and a timestamp. The timestamp generally includes the year, month, day, hour, minute, second, and timezone. There can be more fields, but at a minimum, it's generally a timestamp and some data.

CloudWatch Logs has built-in integrations with many AWS services, including EC2, VPC Flow Logs, Lambda, CloudTrail, Route 53, and many more. Any services that integrate with CloudWatch Logs can store data directly inside the product. Security for this is generally provided by using IAM roles or service roles.

For anything outside AWS, such as logging custom application or OS logs on EC2, you can use the unified CloudWatch agent. I’ve mentioned this before and will be demoing it later in the EC2 section of the course. This is how anything outside of AWS products and services can log data into CloudWatch Logs. So, it’s either AWS service integrations or the unified CloudWatch agent. There is a third way, using development kits for AWS to implement logging into CloudWatch Logs directly into your application, but that tends to be covered in developer and DevOps AWS courses. For now, just remember either AWS service integrations or the unified CloudWatch agent.

CloudWatch Logs are also capable of taking logging data and generating a metric from it, known as a metric filter. Imagine a situation where you have a Linux instance, and one of the operating system log files logs any failed connection attempts via SSH. If this logging information was injected into CloudWatch Logs, a metric filter can scan those logs constantly. Anytime it sees a mention of the failed SSH connection, it can increment a metric within CloudWatch. You can then have alarms based on that metric, and I’ll be demoing that very thing later in the course.

Let’s look at the architecture visually because I'll be showing you how this works in practice in the CloudTrail demo, which will be coming up later in the section. Architecturally, CloudWatch Logs looks like this: It’s a regional service. So, for this example, let’s assume we’re talking about us-east-1.

The starting point is our logging sources, which can include AWS products and services, mobile or server-based applications, external compute services (virtual or physical servers), databases, or even external APIs. These sources inject data into CloudWatch Logs as log events.

Log events consist of a timestamp and a message block. CloudWatch Logs treats this message as a raw block of data. It can be anything you want, but there are ways the data can be interpreted, with fields and columns defined. Log events are stored inside log streams, which are essentially a sequence of log events from the same source.

For example, if you had a log file stored on multiple EC2 instances that you wanted to inject into CloudWatch Logs, each log stream would represent the log file for one instance. So, you’d have one log stream for instance one and one log stream for instance two. Each log stream is an ordered set of log events for a specific source.

We also have log groups, which are containers for multiple log streams of the same type of logging. Continuing the example, we would have one log group containing everything for that log file. Inside this log group would be different log streams, each representing one source. Each log stream is a collection of log events. Every time an item was added to the log file on a single EC2 instance, there would be one log event inside one log stream for that instance.

A log group also stores configuration settings, such as retention settings and permissions. When we define these settings on a log group, they apply to all log streams within that log group. It’s also where metric filters are defined. These filters constantly review any log events for any log streams in that log group, looking for certain patterns, such as an application error code or a failed SSH login. When detected, these metric filters increment a metric, and metrics can have associated alarms. These alarms can notify administrators or integrate with AWS or external systems to take action.

CloudWatch Logs is a powerful product. This is the high-level architecture, but don’t worry—you’ll get plenty of exposure to it throughout the course because many AWS products integrate with CloudWatch Logs and use it to store their logging data. We’ll be coming back to this product time and again as we progress through the course. CloudTrail uses CloudWatch Logs, Lambda uses CloudWatch Logs, and VPC Flow Logs use CloudWatch Logs. There are many examples of AWS products where we’ll be integrating them with CloudWatch Logs.

I just wanted to introduce it at this early stage of the course. That’s everything I wanted to cover in this theory lesson. Thanks for watching. Go ahead, complete this video, and when you’re ready, join me in the next.

Welcome back, and in this demo lesson, I want to give you some experience working with Service Control Policies (SCPs).

At this point, you've created the AWS account structure which you'll be using for the remainder of the course. You've set up an AWS organization, with the general account that created it becoming the management account. Additionally, you've invited the production AWS account into the organization and created the development account within it.

In this demo lesson, I want to show you how you can use SCPs to restrict what identities within an AWS account can do. This is a feature of AWS Organizations.

Before we dive in, let's tidy up the AWS organization. Make sure you're logged into the general account, the management account of the organization, and then navigate to the organization's console. You can either type that into the 'Find Services' box or select it from 'Recently Used Services.'

As discussed in previous lessons, AWS Organizations allows you to organize accounts with a hierarchical structure. Currently, there's only the root container of the organization. To create a hierarchical structure, we need to add some organizational units. We will create a development organizational unit and a production organizational unit.

Select the root container at the top of the organizational structure. Click on "Actions" and then "Create New." For the production organizational unit, name it 'prod.' Scroll down and click on "Create Organizational Unit." Next, do the same for the development unit: select 'Route,' click on "Actions," and then "Create New." Under 'Name,' type 'dev,' scroll down, and click on "Create Organizational Unit."

Now, we need to move our AWS accounts into these relevant organizational units. Currently, the Development, Production, and General accounts are all contained in the root container, which is the topmost point of our hierarchical structure.

To move the accounts, select the Production AWS account, click on "Actions," and then "Move." In the dialogue that appears, select the Production Organizational Unit and click "Move." Repeat this process for the Development AWS account: select the Development AWS account, click "Actions," then "Move," and select the 'dev' OU before clicking "Move."

Now, we've successfully moved the two AWS accounts into their respective organizational units. If you select each organizational unit in turn, you can see that 'prod' contains the production AWS account, and 'dev' contains the development AWS account. This simple hierarchical structure is now in place.

To prepare for the demo part of this lesson where we look at SCPs, move back to the AWS console. Click on AWS, then the account dropdown, and switch roles into the production AWS account by selecting 'Prod' from 'Role History.'

Once you're in the production account, create an S3 bucket. Type S3 into the 'Find Services' box or find it in 'Recently Used Services' and navigate to the S3 console. Click on "Create Bucket." For the bucket name, call it 'CatPics' followed by a random number—S3 bucket names must be globally unique. I’ll use 1, lots of 3s, and then 7. Ensure you select the US East 1 region for the bucket. Scroll down and click "Create Bucket."

After creating the bucket, go inside it and upload some files. Click on "Add Files," then download the cat picture linked to this lesson to your local machine. Upload this cat picture to the S3 bucket by selecting it and clicking "Open," then "Upload" to complete the process.

Once the upload finishes, you can view the picture of Samson. Click on it to see Samson looking pretty sleepy. This demonstrates that you can currently access the Samson.jpg object while operating within the production AWS account.

The key point here is that you’ve assumed an IAM role. By switching roles into the production account, you’ve assumed the role called "organization account access role," which has the administrator access managed policy attached.

Now, we’ll demonstrate how this can be restricted using SCPs. Move back to the main AWS console. Click on the account dropdown and switch back to the general AWS account. Navigate to AWS Organizations, then Policies. Currently, most options are disabled, including Service Control Policies, Tag Policies, AI Services, Opt-out Policies, and Backup Policies.

Click on Service Control Policies and then "Enable" to activate this functionality. This action adds the "Full AWS Access" policy to the entire organization, which imposes no restrictions, so all AWS accounts maintain full access to all AWS services.

To create our own service control policy, download the file named DenyS3.json linked to this lesson and open it in a code editor. This SCP contains two statements. The first statement is an allow statement with an effect of allow, action as star (wildcard), and resource as star (wildcard). This replicates the full AWS access SCP applied by default. The second statement is a deny statement that denies any S3 actions on any AWS resource. This explicit deny overrides the explicit allow for S3 actions, resulting in access to all AWS services except S3.

Copy the content of the DenyS3.json file into your clipboard. Move back to the AWS console, go to the policy section, and select Service Control Policies. Click "Create Policy," delete the existing JSON in the policy box, and paste the copied content. Name this policy "Allow all except S3" and create it.

Now, go to AWS Accounts on the left menu, select the prod OU, and click on the Policies tab. Attach the new policy "Allow all except S3" by clicking "Attach" in the applied policies box. We will also detach the full AWS access policy directly attached. Check the box next to full AWS access, click "Detach," and confirm by clicking "Detach Policy."

Now, the only service control policy directly attached to production is "Allow all except S3," which allows access to all AWS products and services except S3.

To verify, go back to the main AWS console and switch roles into the production AWS account. Go to the S3 console and you should receive a permissions error, indicating that you don't have access to list buckets. This is because the SCP attached to the production account explicitly denies S3 access. Access to other services remains unaffected, so you can still interact with EC2.

If we switch back to the general account, reattach the full AWS access policy, and detach "Allow all except S3," the production account will regain access to S3. By following the same process, you’ll be able to access the S3 bucket and view the object once again.

This illustrates how SCPs can be used to restrict access for identities within an AWS account, in this case, the production AWS account.

To clean up, delete the bucket. Select the catpics bucket, click "Empty," type "permanently delete," and select "Empty." Once that's done, you can delete the bucket by selecting it, clicking "Delete," confirming the bucket name, and then clicking "Delete Bucket."

You’ve now demonstrated full control over S3, evidenced by successfully deleting the bucket. This concludes the demo lesson. You’ve created and applied an SCP that restricts S3 access, observed its effects, and cleaned up. We’ll discuss more about boundaries and restrictions in future lessons. For now, complete this video, and I'll look forward to seeing you in the next lesson.

Welcome back, and in this lesson, I'll be talking about service control policies, or SCPs. SCPs are a feature of AWS Organizations which can be used to restrict AWS accounts. They're an essential feature to understand if you are involved in the design and implementation of larger AWS platforms. We've got a lot to cover, so let's jump in and get started.

At this point, this is what our AWS account setup looks like. We've created an organization for Animals4life, and inside it, we have the general account, which from now on I'll be referring to as the management account, and then two member accounts, so production, which we'll call prod, and development, which we'll be calling dev. All of these AWS accounts are within the root container of the organization. That's to say they aren't inside any organizational units. In the next demo lesson, we're going to be adding organizational units, one for production and one for development, and we'll be putting the member accounts inside their respective organizational units.

Now, let's talk about service control policies. The concept of a service control policy is simple enough. It's a policy document, a JSON document, and these service control policies can be attached to the organization as a whole by attaching them to the root container, or they can be attached to one or more organizational units. Lastly, they can even be attached to individual AWS accounts. Service control policies inherit down the organization tree. This means if they're attached to the organization as a whole, so the root container of the organization, then they affect all of the accounts inside the organization. If they're attached to an organizational unit, then they impact all accounts directly inside that organizational unit, as well as all accounts within OUs inside that organizational unit. If you have nested organizational units, then by attaching them to one OU, they affect that OU and everything below it. If you attach service control policies to one or more accounts, then they just directly affect those accounts that they're attached to.

Now, I mentioned in an earlier lesson that the management account of an organization is special. One of the reasons it's special is that even if the management account has service control policies attached, either directly via an organizational unit, or on the root container of the organization itself, the management account is never affected by service control policies. This can be both beneficial and it can be a limitation, but as a minimum, you need to be aware of it as a security practice. Because the management account can't be restricted using service control policies, I generally avoid using the management account for any AWS resources. It's the only AWS account within AWS Organizations which can't be restricted using service control policies. As a takeaway, just remember that the management account is special and it's unaffected by any service control policies, which are attached to that account either directly or indirectly.

Now, service control policies are account permissions boundaries. What I mean by that is they limit what the AWS account can do, including the Account Root User within that account. I talked earlier in the course about how you can't restrict an Account Root User. And that is true. You can't directly restrict what the Account Root User of an AWS account can do. The Account Root User always has full permissions over that entire AWS account, but with a service control policy, you're actually restricting what the account itself can do, specifically any identities within that account. So you're indirectly restricting the Account Root User because you're reducing the allowed permissions on the account; you're also reducing what the effective permissions on the Account Root User are. This is a really fine detail to understand. You can never restrict the Account Root User. It will always have 100% access to the account, but if you restrict the account, then in effect, you're also restricting the Account Root User.

Now, you might apply a service control policy to prevent any usage of that account outside a known region, for example, us-east-1. You might also apply a service control policy which only allows a certain size of EC2 instance to be used within the account. Service control policies are a really powerful feature for any larger, more complex AWS deployments. The critical thing to understand about service control policies is they don't grant any permissions. Service control policies are just a boundary. They define the limit of what is and isn't allowed within the account, but they don't grant permissions. You still need to give identities within that AWS account permissions to AWS resources, but any SCPs will limit the permissions that can be assigned to individual identities.

You can use service control policies in two ways. You can block by default and allow certain services, which is an allow list. Or you can allow by default and block access to certain services, which is a deny list. The default is a deny list. When you enable SCPs on your organization, AWS applies a default policy called full AWS access. This is applied to the organization and all OUs within that organization. This policy means that in the default implementation, service control policies have no effect since nothing is restricted. As a reminder, service control policies don't grant permissions, but when SCPs are enabled, there is an implicit default deny, just like IAM policies. If you had no initial allow, then everything would be denied. So the default is this full access policy, which essentially means no restrictions. It has the effect of making SCPs a deny list architecture, so you need to add any restrictions that you want to any AWS accounts within the organization. An example is that you could add another policy, such as this one, called DenyS3. This adds a deny policy for the entire S3 set of API operations, effectively denying S3. You need to remember that SCPs don't actually grant any access rights, but they establish which permissions can be granted in an account. The same priority rules apply: deny, allow, deny. Anything explicitly allowed in an SCP is a service which can have access granted to identities within that account, unless there's an explicit deny within an SCP, then a service cannot be granted. Explicit deny always wins. And in the absence of either, if we didn't have this full AWS access policy in place, then there would be an implicit deny, which blocks access to everything.

The benefit of using deny lists is that because your foundation is to allow wildcard access, so all actions on all resources, as AWS extends the amounts of products and services which are available inside the platform, this allow list constantly expands to cover those services, so it's fairly low admin overhead. You simply need to add any services which you want to deny access to via an explicit deny. In certain situations, you might need to be more conscious about usage in your accounts, and that's where you'd use allow lists. To implement allow lists, it's a two-part architecture. One part of it is to remove the AWS full access policy. This means that only the implicit default deny is in place and active, and then you would need to add any services which you want to allow into a new policy. In this case, S3 and EC2. So in this architecture, we wouldn't have this full AWS access. We would be explicitly allowing S3 and EC2 access. So no matter what identity permissions identities in this account are provided with, they would only ever be allowed to access S3 and EC2. This is more secure because you have to explicitly say which services can be allowed access for users in those accounts, but it's much easier to make a mistake and block access to services which you didn't intend to. It's also much more admin overhead because you have to add services as your business requirements dictate. You can't simply have access to everything and deny services you don't want access to. With this type of architecture, you have to explicitly add each and every service which you want identities within the account to be able to access. Generally, I would normally suggest using a deny list architecture because, simply put, it's much lower admin overhead.

Before we go into a demo, I want to visually show you how SCPs affect permissions. This is visually how SCPs impact permissions within an AWS account. In the left orange circle, this represents the different services that have been granted access to identities in an account using identity policies. On the right in red, this represents which services an SCP allows access to. So the SCP states that the three services in the middle and the service on the right are allowed access as far as the SCP is concerned, and the identity policies which were applied to identities within the account, so the orange circle on the left, grant access to four different services: the three in the middle and the one on the left.

Only permissions which are allowed within identity policies in the account and are allowed by a service control policy are actually active. On the right, this access permission has no effect because while it's allowed within an SCP, an SCP doesn't grant access to anything; it just controls what can and can't be allowed by identity policies within that account. Because no identity policy allows access to this resource, then it has no effect. On the left, this particular access permission is allowed within an identity policy, but it's not effectively allowed because it's not allowed within an SCP. So only things which are involved, the identity policy and an SCP, are actually allowed. In this case, this particular access permission on the left has no effect because it's not within a service control policy, so it's denied.

At an associate level, this is what you need to know for the exam. It's just simply understanding that your effective permissions for identities within an account are the overlap between any identity policies and any applicable SCPs. This is going to make more sense if you experience it with a demo, so this is what we're going to do next. Now that you've set up the AWS organization for the Animals4life business, it's time to put some of this into action. So I'm going to finish this lesson here and then in the next lesson, which is a demo, we're going to continue with the practical part of implementing SCPs. So go ahead and complete this video, and when you're ready, I'll look forward to you joining me in the next.

Welcome back! In this demo lesson, you're going to create the AWS account structure which you'll use for the remainder of the course. At this point, you need to log in to the general AWS account. I’m currently logged in as the IAM admin user of my general AWS account, with the Northern Virginia region selected.

You’ll need either two different web browsers or a single web browser like Firefox that supports different sessions because we’ll be logged into multiple AWS accounts at once. The first task is to create the AWS organization. Since I'm logged in to a standard AWS account that isn’t part of an AWS organization, it’s neither a management account nor a member account. We need to move to the AWS Organizations part of the console and create the organization.

To start, go to "Find Services," type "Organizations," and click to move to the AWS Organizations console. Once there, click "Create Organization." This will begin the process of creating the AWS organization and convert the standard account into the management account of the organization. Click on "Create Organization" to complete the process. Now, the general account is the management account of the AWS organization.

You might see a message indicating that a verification email has been sent to the email address associated with the general AWS account. Click the link in that email to verify the address and continue using AWS Organizations. If you see this notification, verify the email before proceeding. If not, you can continue.

Now, open a new web browser or a browser session like Firefox and log in to the production AWS account. Ensure this is a separate session; if unsure, use a different browser to maintain logins to both the management and production accounts. I’ll log in to the IAM admin user of the production AWS account.

With the production AWS account logged in via a separate browser session, copy the account ID for the production AWS account from the account dropdown. Then, return to the browser session with the general account, which is now the management account of the organization. We’ll invite the production AWS account into this organization.

Click on "Add Account," then "Invite Account." Enter either the email address used while signing up or the account ID of the production account. I’ll enter the account ID. If you’re inviting an account you administer, no notes are needed. However, if the account is administered by someone else, you may include a message. After entering the email or account ID, scroll down and click "Send Invitation."

Depending on your AWS account, you might receive an error message about too many accounts within the organization. If so, log a support request to increase the number of allowed accounts. If no error message appears, the invite process has begun.

Next, accept the invite from the production AWS account. Go back to the tab with the general AWS account, move to the Organizations console, and click "Invitations" on the middle left. You should see an overview of all invitations related to the production AWS account. Click "Accept" to complete the process of joining the organization. Now, the production account is a member of the AWS organization.

To verify, return to the general account tab and refresh. You should now see two AWS accounts: the general and the production accounts. Next, I’ll demonstrate how to role switch into the production AWS account, now a member of the organization.

When adding an account to an organization, you can either invite an existing account or create a new one within the organization. If creating a new account, a role is automatically created for role switching. If inviting an existing account, you need to manually add this role.

To do this, switch to the browser or session where you're logged into the production AWS account. Click on the services search box, type IAM, and move to the IAM console to create IAM roles. Click on "Create Role," select "Another AWS Account," and enter the account ID of the general AWS account, which is now the management account.

Copy the account ID of the general AWS account into the account ID box, then click "Next." Attach the "AdministratorAccess" policy to this role. On the next screen, name the role "OrganizationAccountAccessRole" with uppercase O, A, A, and R, and note that "Organization" uses the U.S. spelling with a Z. Click "Create Role."

In the role details, select "Trust Relationships" to verify that the role trusts the account ID of your general AWS account, which allows identities within the general account to assume this role.

Next, switch back to the general AWS account. Copy the account ID for the production AWS account because we will switch into it using role switch. In the AWS console, click on the account dropdown and select "Switch Roles." Paste the production account ID into the account ID box, and enter the role name "OrganizationAccountAccessRole" with uppercase O, A, A, and R.

For the display name, use "Prod" for production, and pick red as the color for easy identification. Click "Switch Role" to switch into the production AWS account. You’ll see the red color and "Prod" display name indicating a successful switch.

To switch back to the general account, click on "Switch Back." In the role history section, you can see shortcuts for switching roles. Click "Prod" to switch back to the production AWS account using temporary credentials granted by the assumed role.

Now, let’s create the development AWS account within our organization. Close the browser window or tab with the production AWS account as it’s no longer needed. Return to the AWS Organizations console, click "Add Account," and then "Create Account." Name the account "Development," following the same naming structure used for general and production accounts.

Provide a unique email address for the development AWS account. Use the same email structure you’ve used for previous accounts, such as "Adrian+TrainingAWSDevelopment" for consistency.

In the box for the role name, use "OrganizationAccountAccessRole" with uppercase O, A, A, and R, and the U.S. spelling. Click "Create" to create the development account. If you encounter an error about too many accounts, you might need to request an increase in the account limit.

The development account will be created within the organization, and this may take a few minutes. Refresh to see the new development account with its own account ID. Copy this account ID for the switch role dialogue.

Click on the account dropdown, select "Switch Roles," and enter the new development account ID. For the role name, use "OrganizationAccountAccessRole" and for the display name, use "Dev" for development with yellow as the color for distinction. Click "Switch Role" to switch into the development AWS account.

In the AWS console, you’ll see the new development account. You can switch directly between the general, production, and development accounts using role switch shortcuts. AWS automatically created the "OrganizationAccountAccessRole" in the development account.

In summary, you now have three AWS accounts: the general AWS account (management account), the production AWS account, and the development AWS account. This completes the account structure for the course. Complete this video, and I'll look forward to seeing you in the next lesson.

Welcome to this lesson, where I'll be introducing AWS Organizations. AWS Organizations is a product that allows larger businesses to manage multiple AWS accounts in a cost-effective way with little to no management overhead.

Organizations is a product that has evolved significantly over the past few years, and it's worthwhile to step through that evolution to understand all of its different features. We’ve got a lot to cover, so let's jump in and get started.

Without AWS Organizations, many large businesses would face the challenge of managing numerous AWS accounts. In the example onscreen, there are four accounts, but I've worked with some larger enterprises with hundreds of accounts and have heard of even more. Without AWS Organizations, each of these accounts would have its own pool of IAM users as well as separate payment methods. Beyond 5 to 10 accounts, this setup becomes unwieldy very quickly.

AWS Organizations is a simple product to understand. You start with a single AWS account, which I'll refer to as a standard AWS account from now on. A standard AWS account is an AWS account that is not part of an organization. Using this standard AWS account, you create an AWS Organization.

It’s important to understand that the organization isn't created within this account; you're simply using the account to create the organization. This standard AWS account that you use to create the organization then becomes the Management Account for the organization. The Management Account used to be called the Master Account. If you hear either of these terms—Management Account or Master Account—just know that they mean the same thing.

This is a key point to understand with regards to AWS Organizations because the Management Account is special for two reasons, which I’ll explain in this lesson. For now, I’ll add a crown to this account to indicate that it’s the Management Account and to help you distinguish it from other AWS accounts.

Using this Management Account, you can invite other existing standard AWS accounts into the organization. Since these are existing accounts, they need to approve the invites to join the organization. Once they do, those Standard Accounts will become part of the AWS Organization.

When standard AWS accounts join an AWS Organization, they change from being Standard Accounts to being Member Accounts of that organization. Organizations have one and only one Management or Master Account and then zero or more Member Accounts.

You can create a structure of AWS accounts within an organization, which is useful if you have many accounts and need to group them by business units, functions, or even the development stage of an application. The structure within AWS Organizations is hierarchical, forming an inverted tree.

At the top of this tree is the root container of the organization. This is just a container for AWS accounts at the top of the organizational structure. Don’t confuse this with the Account Root User, which is the admin user of an AWS account. The organizational root is just a container within an AWS Organization, which can contain AWS accounts, including Member Accounts or the Management Account.

As well as containing accounts, the organizational root can also contain other containers, known as organizational units (OUs). These organizational units can contain AWS accounts, Member Accounts, or the Management Account, or they can contain other organizational units, allowing you to build a complex nested AWS account structure within Organizations.

Again, please don’t confuse the organizational root with the AWS Account Root User. The AWS Account Root User is specific to each AWS account and provides full permissions over that account. The root of an AWS Organization is simply a container for AWS accounts and organizational units and is the top level of the hierarchical structure within AWS Organizations.

One important feature of AWS Organizations is consolidated billing. With the example onscreen now, there are four AWS accounts, each with its own billing information. Once these accounts are added to an AWS Organization, the individual billing methods for the Member Accounts are removed. Instead, the Member Accounts pass their billing through to the Management Account of the organization.

In the context of consolidated billing, you might see the term Payer Account. The Payer Account is the AWS account that contains the payment method for the organization. So, if you see Master Account, Management Account, or Payer Account, know that within AWS Organizations, they all refer to the same thing: the account used to create the organization and the account that contains the payment method for all accounts within the AWS Organization.

Using consolidated billing within an AWS Organization means you receive a single monthly bill contained within the Management Account. This bill covers the Management Account and all Member Accounts of the organization. One bill contains all the billable usage for all accounts within the AWS Organization, removing a significant amount of financial admin overhead for larger businesses. This alone would be worth creating an organization for most larger enterprises.

But it gets better. With AWS, certain services become cheaper the more you use them, and for certain services, you can pay in advance for cheaper rates. When using Organizations, these benefits are pooled, allowing the organization to benefit as a whole from the spending of each AWS account within it.

AWS Organizations also features a service called Service Control Policies (SCPs), which allows you to restrict what AWS accounts within the organization can do. These are important, and I’ll cover them in their own dedicated lesson, which is coming up soon. I wanted to mention them now as a feature of AWS Organizations.

Before we go through a demo where we'll create an AWS Organization and set up the final account structure for this course, I want to cover two other concepts. You can invite existing accounts into an organization, but you can also create new accounts directly within it. All you need is a valid, unique email address for the new account, and AWS will handle the rest. Creating accounts directly within the organization avoids the invite process required for existing accounts.

Using an AWS Organization changes what is best practice in terms of user logins and permissions. With Organizations, you don’t need to have IAM Users inside every single AWS account. Instead, IAM roles can be used to allow IAM Users to access other AWS accounts. We’ll implement this in the following demo lesson. Best practice is to have a single account for logging in, which I’ve shown in this diagram as the Management Account of the organization. Larger enterprises might keep the Management Account clean and have a separate account dedicated to handling logins.

Both approaches are fine, but be aware that the architectural pattern is to have a single AWS account that contains all identities for logging in. Larger enterprises might also have their own existing identity system and may use Identity Federation to access this single identity account. You can either use internal AWS identities with IAM or configure AWS to allow Identity Federation so that your on-premises identities can access this designated login account.

From there, we can use this account with these identities and utilize a feature called role switching. Role switching allows users to switch roles from this account into other Member Accounts of the organization. This process assumes roles in these other AWS accounts. It can be done from the console UI, hiding much of the technical complexity, but it’s important to understand how it works. Essentially, you either log in directly to this login account using IAM identities or use Identity Federation to gain access to it, and then role switch into other accounts within the organization.

I’ll discuss this in-depth as we progress through the course. The next lesson is a demo where you’ll implement this yourself and create the final AWS account structure for the remainder of the course.

Okay, so at this point, it's time for a demo. As I mentioned, you'll be creating the account structure you'll use for the rest of the course. At the start, I demoed creating AWS accounts, including a general AWS account and a production AWS account. In the next lesson, I’ll walk you through creating an AWS Organization using this general account, which will become the Management Account for the AWS Organization. Then, you'll invite the existing production account into the organization, making it a Member Account. Finally, you'll create a new account within the organization, which will be the Development Account.

I’m excited for this, and it’s going to be both fun and useful for the exam. So, go ahead and finish this video, and when you're ready, I look forward to you joining me in the next lesson, which will be a demo.

Welcome back.

In this lesson, I want to continue immediately from the last one by discussing when and where you might use IAM roles. By talking through some good scenarios for using roles, I want to make sure that you're comfortable with selecting these types of situations where you would choose to use an IAM role and where you wouldn't, because that's essential for real-world AWS usage and for answering exam questions correctly.

So let's get started.

One of the most common uses of roles within the same AWS account is for AWS services themselves. AWS services operate on your behalf and need access rights to perform certain actions. An example of this is AWS Lambda. Now, I know I haven't covered Lambda yet, but it's a function as a service product. What this means is that you give Lambda some code and create a Lambda function. This function, when it runs, might do things like start and stop EC2 instances, perform backups, or run real-time data processing. What it does exactly isn't all that relevant for this lesson. The key thing, though, is that a Lambda function, as with most AWS things, has no permissions by default. A Lambda function is not an AWS identity. It's a component of a service, and so it needs some way of getting permissions to do things when it runs. Running a Lambda function is known as a function invocation or a function execution using Lambda terminology.

So anything that's not an AWS identity, this might be an application or a script running on a piece of compute hardware somewhere, needs to be given permissions on AWS using access keys. Rather than hard-coding some access keys into your Lambda function, there's actually a better way. To provide these permissions, we can create an IAM role known as a Lambda execution role. This execution role has a trust policy which trusts the Lambda service. This means that Lambda is allowed to assume that role whenever a function is executed. This role has a permissions policy which grants access to AWS products and services.

When the function runs, it uses the sts:AssumeRole operation, and then the Secure Token Service generates temporary security credentials. These temporary credentials are used by the runtime environment in which the Lambda function runs to access AWS resources based on the permissions the role’s permissions policy has. The code is running in a runtime environment, and it's the runtime environment that assumes the role. The runtime environment gets these temporary security credentials, and then the whole environment, which the code is running inside, can use these credentials to access AWS resources.

So why would you use a role for this? What makes this scenario perfect for using a role? Well, if we didn't use a role, you would need to hard-code permissions into the Lambda function by explicitly providing access keys for that function to use. Where possible, you should avoid doing that because, A, it's a security risk, and B, it causes problems if you ever need to change or rotate those access keys. It's always better for AWS products and services, where possible, to use a role, because when a role is assumed, it provides a temporary set of credentials with enough time to complete a task, and then these are discarded.

For a given Lambda function, you might have one copy running at once, zero copies, 50 copies, a hundred copies, or even more. Because you can't determine this number, because it's unknown, if you remember my rule from the previous lesson, if you don't know the number of principals, if it's multiple or if it's an uncertain number, then it suggests a role might be the most ideal identity to use. In this case, it is the ideal way of providing Lambda with these credentials to use a role and allow it to get these temporary credentials. It's always the preferred option when using AWS services to do something on your behalf; use a role because you don't need to provide any static credentials.

Okay, so let's move on to the next scenario.

Another situation where roles are useful is emergency or out-of-the-usual situations. Here’s a familiar scenario that you might find in a workplace. This is Wayne, and Wayne works in a business's service desk team. This team is given read-only access to a customer's AWS account so that they can keep an eye on performance. The idea is that anything more risky than this read-only level of access is handled by a more senior technical team. We don't want to give Wayne's team long-term permissions to do anything more destructive than this read-only access, but there are always going to be situations which occur when we least want them, normally 3:00 a.m. on a Sunday morning, when a customer might call with an urgent issue where they need Wayne's help to maybe stop or start an instance, or maybe even terminate an EC2 instance and recreate it.

So 99% of the time, Wayne and his team are happy with this read-only access, but there are situations when he needs more. This is a break-glass style situation, which is named after this. The idea of break glass in the physical world is that there is a key for something behind glass. It might be a key for a room that a certain team doesn't normally have access to, maybe it’s a safe or a filing cabinet. Whatever it is, the glass provides a barrier, meaning that when people break it, they really mean to break it. It’s a confirmation step. So if you break a piece of glass to get a key to do something, there needs to be an intention behind it. Anyone can break the glass and retrieve the key, but having the glass results in the action only happening when it's really needed. At other times, whatever the key is for remains locked. And you can also tell when it’s been used and when it hasn’t.

A role can perform the same thing inside an AWS account. Wayne can assume an emergency role when absolutely required. When he does, he'll gain additional permissions based on the role's permissions policy. For a short time, Wayne will, in effect, become the role. This access will be logged and Wayne will know to only use the role under exceptional circumstances. Wayne’s normal permissions can remain at read-only, which protects him and the customer, but he can obtain more if required when it’s really needed. So that’s another situation where a role might be a great solution.

Another scenario when roles come in handy is when you're adding AWS into an existing corporate environment. You might have an existing physical network and an existing provider of identities, known as an identity provider, that your staff use to log into various systems. For the sake of this example, let’s just say that it's Microsoft Active Directory. In this scenario, you might want to offer your staff single sign-on, known as SSO, allowing them to use their existing logins to access AWS. Or you might have upwards of 5,000 accounts. Remember, there’s the 5,000 IAM user limit. So for a corporation with more than 5,000 staff, you can’t offer each of them an IAM user. That is beyond the capabilities of IAM.

Roles are often used when you want to reuse your existing identities for use within AWS. Why? Because external accounts can’t be used directly. You can’t access an S3 bucket directly using an Active Directory account. Remember this fact. External accounts or external identities cannot be used directly to access AWS resources. You can’t directly use Facebook, Twitter, or Google identities to interact with AWS. There is a separate process which allows you to use these external identities, which I’ll be talking about later in the course.

Architecturally, what happens is you allow an IAM role inside your AWS account to be assumed by one of the external identities, which is in Active Directory in this case. When the role is assumed, temporary credentials are generated and these are used to access the resources. There are ways that this is hidden behind the console UI so that it appears seamless, but that's what happens behind the scenes. I'll be covering this in much more detail later in the course when I talk about identity federation, but I wanted to introduce it here because it is one of the major use cases for IAM roles.

Now, why roles are so important when an existing ID provider such as Active Directory is involved is that, remember, there is this 5,000 IAM user limit in an account. So if your business has more than 5,000 accounts, then you can’t simply create an IAM user for each of those accounts, even if you wanted to. 5,000 is a hard limit. It can't be changed. Even if you could create more than 5,000 IAM users, would you actually want to manage 5,000 extra accounts? Using a role in this way, so giving permissions to an external identity provider and allowing external identities to assume this role, is called ID Federation. It means you have a small number of roles to manage and external identities can use these roles to access your AWS resources.

Another common situation where you might use roles is if you're designing the architecture for a popular mobile application. Maybe it's a ride-sharing application which has millions of users. The application needs to store and retrieve data from a database product in AWS, such as DynamoDB. Now, I've already explained two very important but related concepts on the previous screen. Firstly, that when you interact with AWS resources, you need to use an AWS identity. And then secondly, that there’s this 5,000 IAM user limit per account. So designing an application with this many users which needs access to AWS resources, if you could only use IAM users or identities in AWS, it would be a problem because of this 5,000 user limit. It’s a hard limit and it can’t be raised.

Now, this is a problem which can be fixed with a process called Web Identity Federation, which uses IAM roles. Most mobile applications that you’ve used, you might have noticed they allow you to sign in using a web identity. This might be Twitter, Facebook, Google, and potentially many others. If we utilize this architecture for our web application, we can trust these identities and allow these identities to assume an IAM role. This is based on that role’s trust policy. So they can assume that role, gain access to temporary security credentials, and use those credentials to access AWS resources, such as DynamoDB. This is a form of Web Identity Federation, and I'll be covering it in much more detail later in the course.

The use of roles in this situation has many advantages. First, there are no AWS credentials stored in the application, which makes it a much more preferred option from a security point of view. If an application is exploited for whatever reason, there’s no chance of credentials being leaked, and it uses an IAM role which you can directly control from your AWS account. Secondly, it makes use of existing accounts that your customers probably already have, so they don't need yet another account to access your service. And lastly, it can scale to hundreds of millions of users and beyond. It means you don’t need to worry about the 5,000 user IAM limit. This is really important for the exam. There are very often questions on how you can architect solutions which will work for mobile applications. Using ID Federation, so using IAM roles, is how you can accomplish that. And again, I'll be providing much more information on ID Federation later in the course.

Now, one scenario I want to cover before we finish up this lesson is cross-account access. In an upcoming lesson, I’ll be introducing AWS Organizations and you will get to see this type of usage in practice. It’s actually how we work in a multi-account environment. Picture the scenario that's on screen now: two AWS accounts, yours and a partner account. Let’s say your partner organization offers an application which processes scientific data and they want you to store any data inside an S3 bucket that’s in their account. Your account has thousands of identities, and the partner IT team doesn’t want to create IAM users in their account for all of your staff. In this situation, the best approach is to use a role in the partner account. Your users can assume that role, get temporary security credentials, and use those to upload objects. Because the IAM role in the partner account is an identity in that account, using that role means that any objects that you upload to that bucket are owned by the partner account. So it’s a very simple way of handling permissions when operating between accounts.

Roles can be used cross-account to give access to individual resources like S3 in the onscreen example, or you can use roles to give access to a whole account. You’ll see this in the upcoming AWS Organization demo lesson. In that lesson, we’re going to configure it so a role in all of the different AWS accounts that we’ll be using for this course can be assumed from the general account. It means you won’t need to log in to all of these different AWS accounts. It makes multi-account management really simple.

I hope by this point you start to get a feel for when roles are used. Even if you’re a little vague, you will learn more as you go through the course. For now, just a basic understanding is enough. Roles are difficult to understand at first, so you’re doing well if you’re anything but confused at this point. I promise you, as we go through the course and you get more experience, it will become second nature.

So at this point, that’s everything I wanted to cover. Thanks for watching. Go ahead and complete this video, and when you're ready, join me in the next lesson.

Welcome back.

Over the next two lessons, I'll be covering a topic which is usually one of the most difficult identity-related topics in AWS to understand, and that's IAM roles. In this lesson, I'll step through how roles work, their architecture, and how you technically use a role. In the following lesson, I'll compare roles to IAM users and go into a little bit more detail on when you generally use a role, so some good scenarios which fit using an IAM role. My recommendation is that you watch both these lessons back to back in order to fully understand IAM roles.

So let's get started.

A role is one type of identity which exists inside an AWS account. The other type, which we've already covered, are IAM users. Remember the term "principal" that I introduced in the previous few lessons? This is a physical person, application, device, or process which wants to authenticate with AWS. We defined authentication as proving to AWS that you are who you say you are. If you authenticate, and if you are authorized, you can then access one or more resources.

I also previously mentioned that an IAM user is generally designed for situations where a single principal uses that IAM user. I’ve talked about the way that I decide if something should use an IAM user: if I can imagine a single thing—one person or one application—who uses an identity, then generally under most circumstances, I'd select to use an IAM user.

IAM roles are also identities, but they're used much differently than IAM users. A role is generally best suited to be used by an unknown number or multiple principals, not just one. This might be multiple AWS users inside the same AWS account, or it could be humans, applications, or services inside or outside of your AWS account who make use of that role. If you can't identify the number of principals which use an identity, then it could be a candidate for an IAM role. Or if you have more than 5,000 principals, because of the number limit for IAM users, it could also be a candidate for an IAM role.

Roles are also something which is generally used on a temporary basis. Something becomes that role for a short period of time and then stops. The role isn't something that represents you. A role is something which represents a level of access inside an AWS account. It's a thing that can be used, short term, by other identities. These identities assume the role for a short time, they become that role, they use the permissions that that role has, and then they stop being that role. It’s not like an IAM user, where you login and it’s a representation of you, long term. With a role, you essentially borrow the permissions for a short period of time.

I want to make a point of stressing that distinction. If you're an external identity—like a mobile application, maybe—and you assume a role inside my AWS account, then you become that role and you gain access to any access rights that that role has for a short time. You essentially become an identity in my account for a short period of time.

Now, this is the point where most people get a bit confused, and I was no different when I first learned about roles. What's the difference between logging into a user and assuming a role? In both cases, you get the access rights that that identity has.

Before we get to the end of this pair of lessons, so this one and the next, I think it's gonna make a little bit more sense, and definitely, as you go through the course and get some practical exposure to roles, I know it's gonna become second nature.

IAM users can have identity permissions policies attached to them, either inline JSON or via attached managed policies. We know now that these control what permissions the identity gets inside AWS. So whether these policies are inline or managed, they're properly referred to as permissions policies—policies which grant, so allow or deny, permissions to whatever they’re associated with.

IAM roles have two types of policies which can be attached: the trust policy and the permissions policy. The trust policy controls which identities can assume that role. With the onscreen example, identity A is allowed to assume the role because identity A is allowed in the trust policy. Identity B is denied because that identity is not specified as being allowed to assume the role in the trust policy.

The trust policy can reference different things. It can reference identities in the same account, so other IAM users, other roles, and even AWS services such as EC2. A trust policy can also reference identities in other AWS accounts. As you'll learn later in the course, it can even allow anonymous usage of that role and other types of identities, such as Facebook, Twitter, and Google.

If a role gets assumed by something which is allowed to assume it, then AWS generates temporary security credentials and these are made available to the identity which assumed the role. Temporary credentials are very much like access keys, which I covered earlier in the course, but instead of being long-term, they're time-limited. They only work for a certain period of time before they expire. Once they expire, the identity will need to renew them by reassuming the role, and at that point, new credentials are generated and given to the identity again which assumed that role.

These temporary credentials will be able to access whatever AWS resources are specified within the permissions policy. Every time the temporary credentials are used, the access is checked against this permissions policy. If you change the permissions policy, the permissions of those temporary credentials also change.

Roles are real identities and, just like IAM users, roles can be referenced within resource policies. So if a role can access an S3 bucket because a resource policy allows it or because the role permissions policy allows it, then anything which successfully assumes the role can also access that resource.

You’ll get a chance to use roles later in this section when we talk about AWS Organizations. We’re going to take all the AWS accounts that we’ve created so far and join them into a single organization, which is AWS’s multi-account management product. Roles are used within AWS Organizations to allow us to log in to one account in the organization and access different accounts without having to log in again. They become really useful when managing a large number of accounts.

When you assume a role, temporary credentials are generated by an AWS service called STS, or the Secure Token Service. This is the operation that's used to assume the role and get the credentials, so sts .

In this lesson, I focused on the technical aspect of roles—mainly how they work. I’ve talked about the trust policy, the permissions policy, and how, when you assume a role, you get temporary security credentials. In the next lesson, I want to step through some example scenarios of where roles are used, and I hope by the end of that, you’re gonna be clearer on when you should and shouldn’t use roles.

So go ahead, finish up this video, and when you’re ready, you can join me in the next lesson.

Welcome back and welcome to this demo of the functionality provided by IAM Groups.

What we're going to do in this demo is use the same architecture that we had in the IAM users demo—the SALLI user and those two S3 buckets—but we’re going to migrate the permissions that the SALLI user has from the user to a group that SALLI is a member of.

Before we get started, just make sure that you are logged in as the IAM admin user of the general AWS account. As always, you’ll need to have the Northern Virginia region selected.

Attached to this video is a demo files link that will download all of the files you’re going to use throughout the demo. To save some time, go ahead and click on that link and start the file downloading. Once it’s finished, go ahead and extract it; it will create a folder containing all of the files you’ll need as you move through the demo.

You should have deleted all of the infrastructure that you used in the previous demo lesson. So at this point, we need to go ahead and recreate it. To do that, attached to this lesson is a one-click deployment link. So go ahead and click that link. Everything is pre-populated, so you need to make sure that you put in a suitable password that doesn’t breach any password policy on your account. I’ve included a suitable default password with some substitutions, so that should be okay for all common password policies.

Scroll down to the bottom, click on the capabilities checkbox, and then create the stack. That’ll take a few moments to create, so I’m going to pause the video and resume it once that stack creation has completed.

Okay, so that’s created now. Click on Services and open the S3 console in a new tab. This can be a normal tab. Go to the Cat Pics bucket, click Upload, add file, locate the demo files folder that you downloaded and extracted earlier. Inside that folder should be a folder called Cat Pics. Go in there and then select merlin.jpg. Click on Open and Upload. Wait for that to finish.

Once it’s finished, go back to the console, go to Animal Pics, click Upload again, add files. This time, inside the Animal Pics folder, upload thaw.jpg. Click Upload. Once that’s done, go back to CloudFormation, click on Resources, and click on the Sally user. Inside the Sally user, click on Add Permissions, Attach Policies Directly, select the "Allow all S3 except cats" policy, click on Next, and then Add Permissions.

So that brings us to the point where we were in the IAM users demo lesson. That’s the infrastructure set back up in exactly the same way as we left the IAM users demo. Now we can click on Dashboard. You’ll need to copy the IAM signing users link for the general account. Copy that into your clipboard.

You’re going to need a separate browser, ideally, a fully separate browser. Alternatively, you can use a private browsing tab in your current browser, but it’s just easier to understand probably for you at this point in your learning if you have a separate browser window. I’m going to use an isolated tab because it’s easier for me to show you.

You’ll need to paste in this IAM URL because now we’re going to sign into this account using the Sally user. Go back to CloudFormation, click on Outputs, and you’ll need the Sally username. Copy that into your clipboard. Go back to this separate browser window and paste that in. Then, back to CloudFormation, go to the Parameters tab and get the password for the Sally user. Enter the password that you chose for Sally when you created the stack.

Then move across to the S3 console and just verify that the Sally user has access to both of these buckets. The easiest way of doing that is to open both of these animal pictures. We’ll start with Thor. Thor’s a big doggo, so it might take some time for him to load in. There we go, he’s loaded in. And the Cat Pics bucket. We get access denied because remember, Sally doesn’t have access to the Cat Pics bucket. That’s as intended.

Now we’ll go back to our other browser window—the one where we logged into the general account as the IAM admin user. This is where we’re going to make the modifications to the permissions. We’re going to change the permissions over to using a group rather than directly on the Sally user.

Click on the Resources tab first and select Sally to move across to the Sally user. Note how Sally currently has this managed policy directly attached to her user. Step one is to remove that. So remove this managed policy from Sally. Detach it. This now means that Sally has no permissions on S3. If we go back to the separate browser window where we’ve got Sally logged in and then hit refresh, we see she doesn’t have any permissions now on S3.

Now back to the other browser, back to the one where we logged in as IAM admin, click on User Groups. We’re going to create a Developers group. Click on Create New Group and call it Developers. That’s the group name. Then, down at the bottom here, this is where we can attach a managed policy to this group. We’re going to attach the same managed policy that Sally had previously directly on her user—Allow all S3 except cats.

Type "allow" into the filter box and press Enter. Then check the box to select this managed policy. We could also directly at this stage add users to this group, but we’re not going to do that. We’re going to do that as a separate process. So click on 'Create Group'.

So that’s the Developers group created. Notice how there are not that many steps to create a group, simply because it doesn’t offer that much in the way of functionality. Open up the group. The only options you see here are 'User Membership' and any attached permissions. Now, as with a user, you can attach inline policies or managed policies, and we’ve got the managed policy.

What we’re going to do next is click on Users and then Add Users to Group. We’re going to select the Sally IAM user and click on Add User. Now our IAM user Sally is a member of the Developers group, and the Developers group has this attached managed policy that allows them to access everything on S3 except the Cat Pics bucket.

Now if I move back to my other browser window where I’ve got the Sally user logged in and then refresh, now that the Sally user has been added to that group, we’ve got permissions again over S3. If I try to access the Cat Pics bucket, I won’t be able to because that managed policy that the Developers team has doesn’t include access for this. But if I open the Animal Pics bucket and open Thor again—Thor’s a big doggo, so it’ll take a couple of seconds—it will load in that picture absolutely fine.

So there we go, there’s Thor. That’s pretty much everything I wanted to demonstrate in this lesson. It’s been a nice, quick demo lesson. All we’ve done is create a new group called Developers, added Sally to this Developers group, removed the managed policy giving access to S3 from Sally directly, and added it to the Developers group that she’s now a member of. Note that no matter whether the policy is attached to Sally directly or attached to a group that Sally is a member of, she still gets those permissions.

That’s everything I wanted to cover in this demo lesson. So before we finish up, let’s just tidy up our account. Go to Developers and then detach this managed policy from the Developers group. Detach it, then go to Groups and delete the Developers group because it wasn’t created as part of the CloudFormation template.

Then, as the IAM admin user, open up the S3 console. We need to empty both of these buckets. Select Cat Pics, click on Empty. You’ll need to type or copy and paste 'Permanently Delete' into that box and confirm the deletion. Click Exit. Then select the Animal Pics bucket and do the same process. Copy and paste 'Permanently Delete' and confirm by clicking on Empty and then Exit.

Now that we’ve done that, we should have no problems opening up CloudFormation, selecting the IAM stack, and then hitting Delete. Note if you do have any errors deleting this stack, just go into the stack, select Events, and see what the status reason is for any of those deletion problems. It should be fairly obvious if it can’t delete the stack because it can’t delete one or more resources, and it will give you the reason why.

That being said, at this point, assume the stack deletions worked successfully, and we’ve cleaned up our account. That’s everything I wanted to cover in this demo lesson. Go ahead, complete this video, and when you’re ready, I’ll see you in the next lesson.