Menu
AWS Application Discovery Service
User Guide

AWS Application Discovery Service CLI Walkthrough

This tutorial walks you through an example that shows you how to use AWS Application Discovery Service with Amazon EC2 instances. The tutorial uses sample scripts and a sample application that you'll need to download.

This topic includes the following:

Setting Up the Environment

The following procedure shows you how to set up an environment from an Amazon EC2 instance. From this instance, you download and configure the AWS CLI, download and run a setup script, and launch a sample application that you use to access the Application Discovery Service API.

Prerequisites

Verify that you have completed the steps in Setting Up AWS Application Discovery Service. By completing the steps in that topic, you ensure the following:

  • You have an AWS account.

  • Your account has been whitelisted and granted access to Application Discovery Service.

  • You have a key pair named discovery.

  • The IAM policy attached to your user account has access to Application Discovery Service resources, including the Arsenal service.

You can set up either Ubuntu Linux or Microsoft Windows EC2 instances for this tutorial.

Configure an Ubuntu Linux Environment

To configure an Ubuntu Linux environment

  1. Open the Amazon EC2 console and switch to the us-west-2 (Oregon) region.

  2. Launch an Ubuntu 14.04 Amazon Machine Image (AMI) using this ID: ami-966687f6. You must launch this AMI because it includes dependencies required for this tutorial.

  3. Log into the Ubuntu instance. Note that the default user for this AMI is "ubuntu," so your SSH command will look something like this:

    Copy
    ssh -i "<path to key>/<key file name>.pem" ubuntu@<IP address of instance>
    Once logged in, use the following commands to install and update the AWS CLI.

    Copy
    sudo apt-get update sudo apt-get install awscli aws configure

    When prompted, specify your whitelisted AWS credentials and set the default region to us-west-2.

  4. Use the following command to download the training script from an Amazon S3 bucket. The blank space and period (.) at the end of the command are required.

    Copy
    aws s3 cp s3://discovery.training.cli.formal/trainingsetup.sh .
  5. The script performs the following tasks:

    • Installs the dependencies required for the Application Discovery Service sample application to run.

    • Downloads an AWS CloudFormation template and stores it locally in the /tmp folder.

    • Installs the Application Discovery Service sample application as a Python package.

    • Installs sample scripts that use the Application Discovery Service API to implement commonly used search queries as a standalone executable.

    Note

    You can view the source code for the scripts in the following directory: ~/discoverytrainingcli/discoverycli/src/. The source code also serves as an example of how to call the API. A readme document with information about the CLI commands is located in the following directory: ~/discoverytrainingcli/discoverycli/doc/README.

  6. Use the following command to run the training setup script:

    Copy
    sudo bash trainingsetup.sh .

    During the installation, the script prompts you to specify your AWS credentials. The sample application uses these credentials to make calls to Application Discovery Service.

  7. Use the following command to view the sample application Help and verify that the sample application installed correctly:

    Copy
    awsdiscovery –h
    
                            Command to verify CLI installation
  8. Use the following command to provision Topology 1 using AWS CloudFormation templates. Topology 1 includes two clients that communicate with two Web servers but only one database.

    Copy
    aws cloudformation create-stack --stack-name topology1 --template-body file:///tmp/SA_cloud_formation_topology2_vpc_with_key_without_EIP
  9. After the command execution completes, you can view the resources that were created using the following command:

    Copy
    aws cloudformation list-stack-resources --stack-name topology1
  10. Use the following command to provision Topology 2 using AWS CloudFormation templates. Topology 2 includes a client that communicate with a Web server and one database.

    Copy
    aws cloudformation create-stack --stack-name topology2 --template-body file:///tmp/SA_cloud_formation_vpc_with_key_without_EIP
  11. After the command execution completes, you can view the resources that were created using the following command:

    Copy
    aws cloudformation list-stack-resources --stack-name topology2

Configure a Microsoft Windows Environment

To configure a Microsoft Windows environment

  1. Open the Amazon EC2 console and switch to the us-west-2 region.

  2. Launch a Microsoft Windows Server 2012 R2 Base Amazon Machine Image (AMI).

  3. Log onto the Windows instance, then download, and install Python. The sample application for this tutorial is a Python application.

  4. In the Install wizard, scroll down to locate the Add Python.exe to Path option. Choose Will be installed on local hard drive and then complete the wizard.

    
                            Python installation
  5. Open a command prompt and run the following command to install the AWS CLI:

    Copy
    pip install awscli

    If the AWS CLI is already installed, run the following command to update it:

    Copy
    pip install --upgrade awscli
  6. Run the following command to configure the session:

    Copy
    aws configure

    When prompted, specify your AWS credentials and set the default region to us-west-2. The AWS credentials must have been granted access to Application Discovery Service. For more information, see Setting Up AWS Application Discovery Service.

  7. Use the following command to download the training script from an Amazon S3 bucket. The blank space and period (.) at the end of the command are required.

    Copy
    aws s3 cp s3://discovery.training.cli.formal/DiscoveryCli.tgz .

    By default, the command downloads the file to the current user folder. For example, if you run the command as the Administrator, the command downloads the file to the C:\Users\Administrator folder.

  8. Unzip the DiscoveryCli.tgz file using a zip/unzip utility like WinZip or 7-Zip. The extraction places a folder called DiscoveryCli in the specified location. Inside that folder, locate and unzip the DiscoveryCli.tar file.

  9. In a command prompt, run the following command in the directory where the setup.py file was extracted during the most recent unzip. If you chose the defaults, the file is located at C:\Users\user name\DiscoveryCli\DiscoveryCli\DiscoveryCli.

    Copy
    python setup.py install
  10. Run the following command to download files for this tutorial. Replace Username with the name of your Windows account.

    Copy
    aws s3 cp s3://poseidon.service.json/service-2.json c:\Users\Username\.aws\models\
  11. The command performs the following tasks:

    • Installs the dependencies required for the Application Discovery Service sample application to run.

    • Installs the Application Discovery Service sample application as a Python package.

    • Installs sample scripts that use the Application Discovery Service API to implement commonly used search queries as a standalone executable.

    Note

    You can view the source code for the scripts in the following directory: C:\Users\user name\DiscoveryCli\DiscoveryCli\DiscoveryCli\src. The source code also serves as an example of how to call the API. A README document with information about the CLI commands is located in the following directory: C:\Users\user name\DiscoveryCli\DiscoveryCli\DiscoveryCli\doc\README

  12. Use the following command to view the sample application Help and verify that the sample application installed correctly:

    Copy
    python C:\python27\Scripts\awsdiscovery --help
  13. Use the following command to download AWS CloudFormation templates and store them locally in the /tmp folder. This command provisions Topology 1. Topology 1 includes two clients that communicate with two Web servers but only one database.

    Copy
    aws s3 cp s3://sa.cf.template/SA_cloud_formation_vpc_with_key_without_EIP C:\tmp\
  14. Launch the template using the following command:

    Copy
    C:\Users\user name\DiscoveryCli\DiscoveryCli>aws cloudformation create-stack --stack-name topology1 --template-body file://C:/tmp/SA_cloud_formation_vpc_with_key_without_EIP
  15. Use the following command to provision Topology 2. Topology 2 includes a client that will communicate with a Web server and one database.

  16. Launch the template using the following command:

    Copy
    C:\Users\user name\DiscoveryCli\DiscoveryCli>aws cloudformation create-stack --stack-name topology2 --template-body file://C:/tmp/SA_cloud_formation_topology2_vpc_with_key_without_EIP

Managing AWS Application Discovery Agents

After you install the AWS Application Discovery Agent on an Amazon EC2 instance, or on a server or virtual machine (VM) in your data center, you’re ready for the next section of the tutorial. This section shows you how to manage the agents that were installed in the topologies as part of the AWS CloudFormation templates.

Note

Instance IDs throughout this tutorial have been made generic and are shown, for example, in the following format: i-xxxxxxx1.

To manage AWS Application Discovery Agents

  1. Use the following command to get the status and location of installed agents.

    On Linux:

    Copy
    awsdiscovery describe-agents

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery describe-agents

    The command output includes the agent status.

    • UNKNOWN – Application Discovery Service does not know the agent status.

    • HEALTHY – Agents responded to the command, but are not collecting data.

    • RUNNING – Agents are collecting data.

    The following sample output shows that all agents are healthy, but not running.

    
                        Get status and location of installed agents
  2. Use the following command to instruct specific agents to start collecting data.

    On Linux:

    Copy
    awsdiscovery start-data-collection i-xxxxxxx1 i-xxxxxx2 i-xxxxxx3 i-xxxxxx4 i-xxxxxx5 i-xxxxxx6 i-xxxxxx7 i-xxxxxx8

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery start-data-collection i-xxxxxxx1 i-xxxxxx2 i-xxxxxx3 i-xxxxxx4 i-xxxxxx5 i-xxxxxx6 i-xxxxxx7 i-xxxxxx8
    
                        Start data collection with Application Discovery Service
  3. Use the following command to get the agent status after they were instructed to start collecting data.

    On Linux:

    Copy
    awsdiscovery describe-agents

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery describe-agents
    
                        Get agent status

    Application Discovery Service takes several minutes to receive and process the data.

  4. Use the following command to instruct specific agents to stop collecting data.

    On Linux:

    Copy
    awsdiscovery stop-data-collection i-xxxxxxx1 i-xxxxxxx2 i-xxxxxxx3 i-xxxxxxx4 i-xxxxxxx5 i-xxxxxxx6 i-xxxxxxx7 i-xxxxxxx8

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery stop-data-collection i-xxxxxxx1 i-xxxxxxx2 i-xxxxxxx3 i-xxxxxxx4 i-xxxxxxx5 i-xxxxxxx6 i-xxxxxxx7 i-xxxxxxx8
    
                        Get agent status

Working with Configuration Items

A configuration item is an IT asset that was discovered in your data center by an agent. With Application Discovery Service, you can query specific configuration items.

To query specific configuration items

  1. Use the following command to list discovered servers.

    On Linux:

    Copy
    awsdiscovery list-servers

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery list-servers
    
                        Listing discovered servers
  2. Use the following command to list servers running a specific process.

    On Linux:

    Copy
    awsdiscovery list-servers --process_name mysql

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery list-servers --process_name mysql
    
                        Listing servers running a specific process
  3. Use the following command to list a server by its MAC address and display detailed attributes.

    On Linux:

    Copy
    awsdiscovery list-servers --mac 06:22:14:9c:96:f9 –verbose

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery list-servers --mac 06:22:14:9c:96:f9 –verbose
    
                        Listing a server by its MAC address

Query Server Dependencies and Connections

This section shows you how to query configuration items based on the following discovery objectives. In this case, the user wants to:

  • Inspect server dependencies based on network connectivity.

  • Identify the processes in a server that were responsible for the connection.

To query configuration items

  1. Use the following command to list all connections.

    On Linux:

    Copy
    awsdiscovery list-connections

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery list-connections
    
                        Listing connections

    Output in the image has been truncated.

  2. Use the following command to list all dependencies for a specific host.

    On Linux:

    Copy
    awsdiscovery list-connections --src_hostname ip-10-0-0-109

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery list-connections --src_hostname ip-10-0-0-109
    
                        Listing connections for a specific host

    Output in the image has been truncated.

  3. Use the following command to list all dependents for a specific host.

    On Linux:

    Copy
    awsdiscovery list-connections --dest_hostname ip-10-0-0-109

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery list-connections --dest_hostname ip-10-0-0-109
    
                        Listing dependents for a specific host

    Output in the image has been truncated.

  4. Use the following command to list Linux-uwsgi servers communicating with Ubuntu-MySQL servers.

    On Linux:

    Copy
    aluwsgi_ubmysql

    On Windows

    Copy
    python C:\python27\Scripts\aluwsgi_ubmysql
    
                        Listing Linux-uwsgi servers communicating with Ubuntu-MySQL
                            servers
  5. Use the following command to list Linux-uwsgi servers communicating with Ubuntu-MySQL servers.

    On Linux:

    Copy
    aluwsgi_ubmysql

    On Windows:

    Copy
    python C:\python27\Scripts\aluwsgi_ubmysql
    
                        Listing Linux-uwsgi servers communicating with Ubuntu-MySQL
                            servers

Working with Multi-Layered Stacks

The previous examples focused on discovery in a two-layered stack (for example, a server communicating with a database). This section shows you how to query configuration items for a three-layered stack. In the samples below, the stack includes a client computer that communicates with an NGINX web server that communicates with a MySQL database.

To query configuration items for a three-layered stack

  • Use the following command to list three-layer stacks.

    On Linux:

    Copy
    three_layered_stacks

    On Windows:

    Copy
    python C:\python27\Scripts\three_layered_stacks

                Listing three-layered stacks

Rows one and two of the command output reveal two clients communicating with two Web servers but only one database. The topology is as follows:


                Listing three-layered stacks

Row three of the command output reveals another three-layered stack. The topology is as follows:


                Listing three-layered stacks

Tag and Export Configuration Items

You can tag discovered configuration items. Tags are metadata that help you categorize IT assets in your data center. Tags use a key,value format. You can also export configuration items and tagged items to a database or an Amazon S3 bucket as a CSV file.

To tag and export configuration items

  1. Use the following command to tag servers. The output shows if the configuration item was tagged.

    On Linux:

    Copy
    awsdiscovery create-tag --hostname ip-10-0-1-166 --tag key=serverType,value=webServer

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery create-tag --hostname ip-10-0-1-166 --tag key=serverType,value=webServer
  2. Use the following command to list tagged servers.

    On Linux:

    Copy
    awsdiscovery list-servers --tag key=serverType,value=webserver

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery list-servers --tag key=serverType,value=webserver
    
                        Listing tagged servers
  3. Use the following command to delete a tag. The output shows if the configuration item was untagged.

    On Linux:

    Copy
    awsdiscovery delete-tag –tag --hostname ip-10-0-1-166 --tag key=serverType,value=webServer

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery delete-tag –tag --hostname ip-10-0-1-166 --tag key=serverType,value=webServer
  4. Use the following command to export data. The output includes an export ID which you use to view the export status.

    On Linux:

    Copy
    awsdiscovery create-export-task

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery create-export-task
  5. Use the following command to view the export status. The output returns the status of the export and a URL where you can view the data.

    On Linux:

    Copy
    awsdiscovery describe-export-tasks export-xxxxxx-xxxx-xxxx-xxxx-6ed91c5c014e

    On Windows:

    Copy
    python C:\python27\Scripts\awsdiscovery describe-export-tasks export-xxxxxx-xxxx-xxxx-xxxx-6ed91c5c014e