If you're using AWS in an enterprise environment and trying to follow best practices, you might be using a multi-account strategy. AWS Organizations is the main tool to achieve this, allowing you to centrally manage and easily govern the dozens of accounts that you will need.

It is however still tricky to list all your AWS resources among all your accounts at the Organization level. Until Amazon comes up with a new service called 'AWS Inventory', we need to write custom scripts to do that for us.

This guide will describe how we can list all EC2 instances within our AWS organization using Golang.

Requirements

  • Go 1.15.X
  • An AWS organization already created. Your personal user/role need the permission to create Roles in IAM

IAM

IAM

In order to make the API calls in each account, you will need a role deployed at the organization level that your user can assume — e.g. OrganizationEc2ReadRole. This role must have read permissions for EC2 and one way to achieve this is to attach the right AWS Managed policy:

arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess

Additionally, this same role will need organization permissions in the master Organization account, so we are able to automatically retrieve all the accounts IDs within our Org. This will be useful as our organization might grow in the future. You can use the AWS managed policy:

arn:aws:iam::aws:policy/AWSOrganizationsReadOnlyAccess

Don't forget to add a trust relationship policy to the role, where {account_id} is the account where your user is configured.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::{account_id}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

When you have this role deployed in all your accounts, it is then easy to allow your user to assume it at the Organization level with a policy using a wildcard:

{
 "Version": "2012-10-17",
 "Statement": [{
  "Effect": "Allow",
  "Principal": {
   "AWS": "arn:aws:iam::*:role/OrganizationEc2ReadRole"
  },
  "Action": "sts:AssumeRole"
 }]
}

Let's code

This blog post explains clearly how to assume roles using go and the code we will write in assume.go is based on it. In short, we are creating a custom struct to store a temporary session and additional configuration including the region, account id and role.

// Clients Struct to store the session with custom parameters
type Clients struct {
	session *session.Session
	configs map[string]*aws.Config
}

This allows us to dynamically create the EC2 and organization clients:

// Organization Create client
func (c *Clients) Organization(
	region string,
	accountID string,
	role string) *organizations.Organizations {
	return organizations.New(c.Session(), c.Config(&region, &accountID, &role))
}

// EC2 Create client
func (c *Clients) EC2(
	region string,
	accountID string,
	role string) *ec2.EC2 {
	return ec2.New(c.Session(), c.Config(&region, &accountID, &role))
}

The complete assume.go file should contain the following code:

// Cross account logic, forked from https://maori.geek.nz/assuming-roles-in-aws-with-go-aeeb28fab418
package main

import (
	"fmt"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials/stscreds"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/ec2"
	"github.com/aws/aws-sdk-go/service/organizations"
)

// Clients Struct to store the session with custom parameters
type Clients struct {
	session *session.Session
	configs map[string]*aws.Config
}

// Session Func to start a session
func (c Clients) Session() *session.Session {
	if c.session != nil {
		return c.session
	}
	sess := session.Must(session.NewSession())
	c.session = sess
	return sess
}

// Config custom func
func (c Clients) Config(
	region *string,
	accountID *string,
	role *string) *aws.Config {

	// return no config for nil inputs
	if accountID == nil || region == nil || role == nil {
		return nil
	}
	arn := fmt.Sprintf(
		"arn:aws:iam::%v:role/%v",
		*accountID,
		*role,
	)
	// include region in cache key otherwise concurrency errors
	key := fmt.Sprintf("%v::%v", *region, arn)

	// check for cached config
	if c.configs != nil && c.configs[key] != nil {
		return c.configs[key]
	}
	// new creds
	creds := stscreds.NewCredentials(c.Session(), arn)
	// new config
	config := aws.NewConfig().
		WithCredentials(creds).
		WithRegion(*region).
		WithMaxRetries(10)
	if c.configs == nil {
		c.configs = map[string]*aws.Config{}
	}
	c.configs[key] = config
	return config
}

// Organization Create client
func (c *Clients) Organization(
	region string,
	accountID string,
	role string) *organizations.Organizations {
	return organizations.New(c.Session(), c.Config(&region, &accountID, &role))
}

// EC2 Create client
func (c *Clients) EC2(
	region string,
	accountID string,
	role string) *ec2.EC2 {
	return ec2.New(c.Session(), c.Config(&region, &accountID, &role))
}

Feel free to read the blog post I mentioned for more details.

Setting up variables

To make our deployment more flexible, I wanted to inject variables from a JSON configuration file. I am using this module that will process the file into a custom object Config that we define. This is the complete config/init.go file:

// Initialize the variables from the json confile file
// Forked from https://github.com/tkanos/gonfig

package config

import (
	"github.com/tkanos/gonfig"
	"fmt"
)

// Config structure
type Config struct {
	region string
	organizationRole string
	masterAccountID string
}

// InitVariables based on json file
func InitVariables(input ...string) Config {
	configuration := Config{}
	fileName := fmt.Sprintf("./config/default.json")
	gonfig.GetConf(fileName, &configuration)

	return configuration
}

We will define our variables in config/default.json. Populate it with the AWS region you're using and the cross-account role mentioned in the IAM part of this guide. You also need to provide the Master account ID as this is the only account number we can't retrieve programmatically.

{
  "Region": "{aws_region}",
  "OrganizationRole": "{aws_role}",
  "MasterAccountID": "{account_id}"
}

Getting the accounts numbers

We like automating everything here. Everything. Therefore, we will use the AWS API to retrieve the list of all the accounts within our organization. This will be extremely useful if you're managing dozens of accounts.

We will write the code that makes API calls in aws.go. The first thing to do is to create a session using our package from assume.go:

// Create organization service client
var c Clients
svc := c.Organization(config.Region, config.MasterAccountID, config.OrganizationRole)

We can then use the *Organizations.ListAccounts call. A default input object needs to be passed (see docs for more info).

// Create variable for the list of accounts and initialize input
organizationAccounts := make(map[string]string)
input := &organizations.ListAccountsInput{}

The maximum amount of results per call being 20, you can use a for loop to iterate over the next pages until all the accounts have been retrieved. AWS provides a NextToken in the result of the call if there are more accounts to display, so we can break the loop when NextToken is null.

// Check if more accounts need to be retrieved using the api token, otherwise break the loop
if organizationAccountsPaginated.NextToken == nil {
	break
} else {
	input = &organizations.ListAccountsInput{NextToken: organizationAccountsPaginated.NextToken}
}

In each iteration of the loop, we will add the result of the current page in a variable called OrganizationAccounts defined outside of the loop scope. This is what the complete function will look like:

// Retrieve all accounts within organization
func getOrganizationAccounts(config config.Config) map[string]string {
	// Create organization service client
	var c Clients
	svc := c.Organization(config.Region, config.MasterAccountID, config.OrganizationRole)
	// Create variable for the list of accounts and initialize input
	organizationAccounts := make(map[string]string)
	input := &organizations.ListAccountsInput{}
	// Start a do-while loop
	for {
		// Retrieve the accounts with a limit of 20 per call
		organizationAccountsPaginated, err := svc.ListAccounts(input)
		// Append the accounts from the current call to the total list
		for _, account := range organizationAccountsPaginated.Accounts {
			organizationAccounts[*account.Name] = *account.Id
		}
		checkError("Could not retrieve account list", err)
		// Check if more accounts need to be retrieved, otherwise break the loop
		if organizationAccountsPaginated.NextToken == nil {
			break
		} else {
			input = &organizations.ListAccountsInput{NextToken: organizationAccountsPaginated.NextToken}
		}
	}
	return organizationAccounts
}

Note that I am using a helper function defined in helpers.go to handle errors in my code. This is optional, feel free to use your own solution to check errors.

// Function that log errors if not null
func checkError(message string, err error) {
	if err != nil {
		log.Fatal(message, err)
	}
}

You can already execute the code you have so far from the command line. Create main.go with the following:

// Main file
package main

import (
	"./config" // Import config file
	"fmt"
)

func main() {
	// Retrieve config file
	config := config.InitVariables()

	// Print all accounts names & ID from the organization
	fmt.Println(getOrganizationAccounts(config))
}

After exporting your AWS credentials and fill the config file, you should be able to run go run . and output all the accounts in your terminal. Make sure this is working correctly before moving on to the next step!

Retrieving all EC2 instances within one account

It's time to write the main function of our code: getAccountEc2. It will be responsible of starting a new EC2 client, listing the EC2 instances of a given account, and parse the result in a map slice with the attributes that we need, e.g. Instance Name, ID, Image ID, Private IP adress etc.

The call that we are making is *Ec2.DescribeInstances

// Create EC2 service client
var c Clients
svc := c.EC2(config.Region, accountID, config.OrganizationRole)
// Get the EC2 list of the given account
input := &ec2.DescribeInstancesInput{}
instances, err := svc.DescribeInstances(input)
checkError("Could not retrieve the EC2s", err)

We can then iterate over the result Ec2 object using a nested loop if the account has more than 0 instances. The parsing part of the function is pretty straightforward: we are appending the attributes to the slice as we find them. The complete function in aws.go  should look like this:

// Retrieve all ec2 instances and their attributes within an account
func getAccountEc2(config config.Config, accountName string, accountID string, result map[string][]string) map[string][]string {
	// Create EC2 service client
	var c Clients
	svc := c.EC2(config.Region, accountID, config.OrganizationRole)
	// Get the EC2 list of the given account
	input := &ec2.DescribeInstancesInput{}
	instances, err := svc.DescribeInstances(input)
	checkError("Could not retrieve the EC2s", err)

	// Iterate over the EC2 instances and add elements to global list, if instances > 0
	if len(instances.Reservations) != 0 {
		for _, reservation := range instances.Reservations {
			// Loop through every individual EC2 instance
			for _, instance := range reservation.Instances {
				// Set the map key using the unique instance ID
				key := *instance.InstanceId
				// Retrieve account information
				result[key] = append(result[key], accountName)
				result[key] = append(result[key], accountID)
				// Check if the instance name is set using tags, otherwise use default null name
				for _, tag := range instance.Tags {
					if *tag.Key == "Name" {
						result[key] = append(result[key], *tag.Value)

					}
				}
				if len(result) == 2 {
					result[key] = append(result[key], "N/A")
				}
				// Retrieve instance information, some use default values  if potentially null
				result[key] = append(result[key], *instance.InstanceType)
				result[key] = append(result[key], *instance.InstanceId)
				result[key] = append(result[key], *instance.ImageId)
				if instance.Platform != nil {
					result[key] = append(result[key], *instance.Platform)
				} else {
					result[key] = append(result[key], "linux")
				}
				if instance.PrivateIpAddress != nil {
					result[key] = append(result[key], *instance.PrivateIpAddress)
				} else {
					result[key] = append(result[key], "N/A")
				}
				result[key] = append(result[key], *instance.State.Name)
				result[key] = append(result[key], (*instance.LaunchTime).String())
			}
		}
	}
	fmt.Println("Account number " + accountID + " done")
	return result
}

Putting It Together

Now that we have our 2 main functions, the only thing left is to make them work together. Using getOrganizationAccounts, we will iterate over each account and describe the EC2 instances with getAccountEc2. Let's complete main.go:

func main() {
	// Retrieve config file
	config := config.InitVariables()

	// Get all accounts names & ID from the organization
	listAccounts := getOrganizationAccounts(config)

	// Create list variable to store every ec2 instances
	var listEc2 = make(map[string][]string)

	// Loop over each account and get its instances via a function
	fmt.Println("Retrieving the instances...")
	for accountName, accountID := range listAccounts {
		listEc2 = getAccountEc2(config, accountName, accountID, listEc2)
	}
	fmt.Println("All the instances from the Organization were retrieved.")

}

And we are done! All our organization instances information is now stored in a map of slices called listEc2. It's now up to you to decide what to do with it.

This is what our final tree looks like:

.
├── assume.go
├── aws.go
├── config
│   ├── default.json
│   └── init.go
├── helpers.go
└── main.go

Going further

In my specific use case, I wanted to print all this data in a CSV file so it could be shared with other engineers within my team. To do this, you can add a new function called writeToCSV in helpers.go and use the csv package to write to a new file with proper headers:

// Function that writes a map of slices to a CSV File
func writeToCSV(listEc2 map[string][]string) {
	// Create the csv file using the os package
	fmt.Println("Creating a CSV file...")
	file, err := os.Create("result.csv")
	checkError("Cannot create file", err)
	defer file.Close()
	// Create the writer object
	writer := csv.NewWriter(file)
	defer writer.Flush()
	// Write headers
	var headers = []string{"Account Name", "Account ID", "Instance Name", "Instance Size", "Instance ID", "Image ID", "Platform", "Private IP", "State", "Timestamp"}
	writer.Write(headers)
	// Loop over the organization ec2 list and write them in rows in the csv file
	for _, value := range listEc2 {
		err := writer.Write(value)
		checkError("Cannot write to file", err)
	}
	fmt.Println("CSV file created in " + "result.csv")
}

You now just have to call this function at the end of your main function.

// Write results to a CSV file
writeToCSV(listEc2)

Drop a comment if you have any questions/issues!

Source code for this guide: https://github.com/FlorianValery/aws-organization-ec2-list