how.wtf

API Key Authentication with API Gateway using AWS CDK

Thomas Taylor — Sun, 30 Jun 2024 17:35:00 -0400

API key authentication is a common method for securing APIs by controlling access to them. It’s important to note that API keys are great for authentication, but further development should be made to ensure proper authorization at the business level. API keys do not ensure that the correct permissions are being enforced, only that the user has access to the API.

Regardless, let’s get started! In this post, we’re going to touch on a few services:

API Gateway with Proxy Integration
Lambda Authorizer
AWS CDK

Get started

Conceptually, the flow of our application will look like this:

Client makes a request to API Gateway with API key
The lambda authorizer determines if the API key is valid
If the API key is valid, the policy is generated and the request is allowed to pass through to the lambda function
If the API key is invalid, the request is denied
The lambda function is invoked and returns a response

Set up the CDK project

Firstly, let’s create the CDK project. I will choose TypeScript as the language, but you can choose any language you prefer. Please refer to the AWS CDK hello world documentation for other supported languages.

1cdk init --language typescript

Next, let’s install the necessary dependencies:

1npm i

In addition, install the @types/aws-lambda package:

1npm i @types/aws-lambda

Let’s start by finding the primary stack file which is located under the lib directory. In my case, it’s lib/api-key-gateway-stack.ts.

Edit the CDK stack

Luckily, in a few lines of code, we can spin up a full-featured API Gateway with a lambda handler using the AWS CDK.

 1import { Duration, Stack, StackProps } from "aws-cdk-lib";
 2import { Construct } from "constructs";
 3import { Runtime } from "aws-cdk-lib/aws-lambda";
 4import {
 5  LambdaRestApi,
 6  TokenAuthorizer,
 7  AuthorizationType,
 8} from "aws-cdk-lib/aws-apigateway";
 9import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
10
11export class ApiKeyGatewayStack extends Stack {
12  constructor(scope: Construct, id: string, props?: StackProps) {
13    super(scope, id, props);
14    const fn = new NodejsFunction(this, "server", {
15      entry: "bin/server.ts",
16      handler: "handler",
17      runtime: Runtime.NODEJS_20_X,
18      timeout: Duration.minutes(1),
19    });
20    const auth = new NodejsFunction(this, "auth", {
21      entry: "bin/auth.ts",
22      handler: "handler",
23      runtime: Runtime.NODEJS_20_X,
24      timeout: Duration.seconds(10),
25    });
26    const api = new LambdaRestApi(this, "api", {
27      handler: fn,
28      defaultMethodOptions: {
29        authorizationType: AuthorizationType.CUSTOM,
30        authorizer: new TokenAuthorizer(this, "authorizer", {
31          handler: auth,
32        }),
33      },
34    });
35  }
36}

Let’s break down the code:

The first construct, NodejsFunction, is a node lambda function that will serve as our primary handler.
The second construct, another NodejsFunction, is a lambda authorizer that will be used to validate the API key.
The third construct, LambdaRestApi, is the API Gateway that includes the first construct wired as the proxy integration and the second construct as the authorizer.

Create the lambda handler

Located at bin/server.ts, we will create a simplistic lambda function that returns Hello, World!.

 1import { APIGatewayProxyEvent, APIGatewayProxyResult } from "aws-lambda";
 2
 3export const handler = async (
 4  event: APIGatewayProxyEvent,
 5): Promise<APIGatewayProxyResult> => {
 6  return {
 7    statusCode: 200,
 8    body: JSON.stringify({ message: "Hello, World!" }),
 9  };
10};

Create the lambda authorizer

Next, let’s create the lambda authorizer located at bin/auth.ts. This lambda function will be responsible for validating the API key.

To keep it simple, we will hardcode the API key to Bearer abc123.

 1import { APIGatewayTokenAuthorizerEvent, Handler } from "aws-lambda";
 2
 3export const handler: Handler = async (
 4  event: APIGatewayTokenAuthorizerEvent,
 5) => {
 6  const effect = event.authorizationToken == "Bearer abc123" ? "Allow" : "Deny";
 7  return {
 8    principalId: "abc123",
 9    policyDocument: {
10      Version: "2012-10-17",
11      Statement: [
12        {
13          Action: "execute-api:Invoke",
14          Effect: effect,
15          Resource: [event.methodArn],
16        },
17      ],
18    },
19  };
20};

Deploy the stack

Now that we have our stack and lambda handlers setup, let’s deploy the stack!

1npx cdk deploy

Once the deployment is complete, you should see the API Gateway endpoint as an output.

 1Do you wish to deploy these changes (y/n)? y
 2ApiKeyGatewayStack: deploying... [1/1]
 3ApiKeyGatewayStack: creating CloudFormation changeset...
 4
 5 ✅  ApiKeyGatewayStack
 6
 7✨  Deployment time: 45.34s
 8
 9Outputs:
10ApiKeyGatewayStack.apiEndpoint9349E63C = https://x2s65m7xyd.execute-api.us-east-1.amazonaws.com/prod/
11Stack ARN:
12arn:aws:cloudformation:us-east-1:123456789012:stack/ApiKeyGatewayStack/0ca225a0-3727-11ef-ae64-0affd17461c9
13✨  Total time: 117.33s

Test the API

Let’s use curl to test the API without the API key.

1curl https://.execute-api.us-east-1.amazonaws.com/prod/

Output:

1{"message":"Unauthorized"}

As expected, we received an unauthorized response. Now, let’s test the API with the API key.

1curl https://x2s65m7xyd.execute-api.us-east-1.amazonaws.com/prod/ \
2    -H "Authorization: Bearer abc123"

Output:

1{"message":"Hello, World!"}

Great! We have successfully created an API Gateway with a lambda authorizer using the AWS CDK. At this point, you may choose to extend the Lambda Authorizer to query another data source like DynamoDB that stores API keys.

Clean up

Lastly, let’s clean up our AWS resources by destroying the stack:

1npx cdk destroy

That’s it! You successfully created an API Gateway with a lambda authorizer using the AWS CDK.

A step-by-step guide on how to use the Amazon Bedrock Converse API

Thomas Taylor — Sun, 02 Jun 2024 13:25:00 -0400

On May 30th, 2024, Amazon announced the release of the Bedrock Converse API. This API is designed to provide a consistent experience for “conversing” with Amazon Bedrock models.

The API supports:

Conversations with multiple turns
System messages
Tool use
Image and text input

In this post, we’ll walk through how to use the Amazon Bedrock Converse API with the Claude Haiku foundational model.

Please keep in mind that not all foundational models may support all the features of the Converse API. For more details per model, see the Amazon Bedrock documentation.

Getting started

To get started, let’s install the boto3 package.

1pip install boto3

Next, we’ll create a new Python script and import the necessary dependencies.

1import boto3
2
3client = boto3.client("bedrock-runtime")

Step 1 - Starting a conversation

To start off simple, let’s send a single message to Claude.

 1import boto3
 2
 3client = boto3.client("bedrock-runtime")
 4
 5messages = [{"role": "user", "content": [{"text": "What is your name?"}]}]
 6
 7response = client.converse(
 8    modelId="anthropic.claude-3-haiku-20240307-v1:0",
 9    messages=messages,
10)
11
12print(response)

The response is a dictionary containing the response and other metadata information from the API. For more information regarding the output, please refer to the Amazon Bedrock documentation.

For the purposes of this post, I’ll print the response as JSON.

 1{
 2  "ResponseMetadata": {
 3    "RequestId": "6984dcf2-c6aa-4000-a3d6-22e34a43df12",
 4    "HTTPStatusCode": 200,
 5    "HTTPHeaders": {
 6      "date": "Sun, 02 Jun 2024 14:54:08 GMT",
 7      "content-type": "application/json",
 8      "content-length": "222",
 9      "connection": "keep-alive",
10      "x-amzn-requestid": "6984dcf2-c6aa-4000-a3d6-22e34a43df12"
11    },
12    "RetryAttempts": 0
13  },
14  "output": {
15    "message": {
16      "role": "assistant",
17      "content": [
18        {
19          "text": "My name is Claude. It's nice to meet you!"
20        }
21      ]
22    }
23  },
24  "stopReason": "end_turn",
25  "usage": {
26    "inputTokens": 12,
27    "outputTokens": 15,
28    "totalTokens": 27
29  },
30  "metrics": {
31    "latencyMs": 560
32  }
33}

To retrieve the contents of the message, we can access the output key in the response.

 1import boto3
 2
 3client = boto3.client("bedrock-runtime")
 4
 5messages = [{"role": "user", "content": [{"text": "What is your name?"}]}]
 6
 7response = client.converse(
 8    modelId="anthropic.claude-3-haiku-20240307-v1:0",
 9    messages=messages,
10)
11
12ai_message = response["output"]["message"]
13output_text = ai_message["content"][0]["text"]
14print(output_text)

Output:

1My name is Claude. It's nice to meet you!

Step 2 - Continuing a conversation

Let’s continue the conversation by appending the AI’s message to the original list of messages. This will allow us to have a multi-turn conversation.

 1import boto3
 2
 3client = boto3.client("bedrock-runtime")
 4
 5messages = [{"role": "user", "content": [{"text": "What is your name?"}]}]
 6
 7response = client.converse(
 8    modelId="anthropic.claude-3-haiku-20240307-v1:0",
 9    messages=messages,
10)
11
12ai_message = response["output"]["message"]
13messages.append(ai_message)
14
15# Let's ask another question
16
17messages.append({"role": "user", "content": [{"text": "Can you help me?"}]})
18response = client.converse(
19    modelId="anthropic.claude-3-haiku-20240307-v1:0",
20    messages=messages,
21)
22
23print(response["output"]["message"]["content"][0]["text"])

Output:

1Yes, I'd be happy to try and help you with whatever you need assistance with. What can I help you with?

Step 3 - Using images

The Amazon Bedrock Converse API supports images as input. Let’s send an image to Claude and see how it responds. I’ll download an image of a cat from Wikipedia and send it to claude.

For this example, I used the requests library to download the image. If you don’t have it installed, you can install it using pip install requests.

1pip install requests

 1import boto3
 2import requests
 3
 4client = boto3.client("bedrock-runtime")
 5
 6messages = [{"role": "user", "content": [{"text": "What is your name?"}]}]
 7
 8response = client.converse(
 9    modelId="anthropic.claude-3-haiku-20240307-v1:0",
10    messages=messages,
11)
12
13ai_message = response["output"]["message"]
14messages.append(ai_message)
15
16messages.append({"role": "user", "content": [{"text": "Can you help me?"}]})
17response = client.converse(
18    modelId="anthropic.claude-3-haiku-20240307-v1:0",
19    messages=messages,
20)
21ai_message = response["output"]["message"]
22messages.append(ai_message)
23
24image_bytes = requests.get(
25    "https://upload.wikimedia.org/wikipedia/commons/4/4d/Cat_November_2010-1a.jpg"
26).content
27messages.append(
28    {
29        "role": "user",
30        "content": [
31            {"text": "What is in this image?"},
32            {"image": {"format": "jpeg", "source": {"bytes": image_bytes}}},
33        ],
34    }
35)
36response = client.converse(
37    modelId="anthropic.claude-3-haiku-20240307-v1:0",
38    messages=messages,
39)
40
41ai_message = response["output"]["message"]
42print(ai_message)

Output:

1{
2  "role": "assistant",
3  "content": [
4    {
5      "text": "The image shows a domestic cat. The cat appears to be a tabby cat with a striped coat pattern. The cat is sitting upright and its green eyes are clearly visible, with a focused and alert expression. The background suggests an outdoor, snowy environment, with some blurred branches or vegetation visible behind the cat."
6    }
7  ]
8}

As you can see, the AI was able to identify the image as a cat and provide a detailed description of the image within a conversational context.

Step 4 - Using a single tool

For this section, let’s start a new conversation with Claude and provide tools it can use.

 1import boto3
 2
 3client = boto3.client("bedrock-runtime")
 4
 5tools = [
 6    {
 7        "toolSpec": {
 8            "name": "get_weather",
 9            "description": "Get the current weather in a given location",
10            "inputSchema": {
11                "json": {
12                    "type": "object",
13                    "properties": {
14                        "location": {
15                            "type": "string",
16                            "description": "The city and state, e.g. San Francisco, CA",
17                        },
18                        "unit": {
19                            "type": "string",
20                            "enum": ["celsius", "fahrenheit"],
21                            "description": "The unit of temperature, either 'celsius' or 'fahrenheit'",
22                        },
23                    },
24                    "required": ["location"],
25                }
26            },
27        }
28    },
29]
30messages = [
31    {
32        "role": "user",
33        "content": [{"text": "What is the weather like right now in New York?"}],
34    }
35]
36response = client.converse(
37    modelId="anthropic.claude-3-haiku-20240307-v1:0",
38    messages=messages,
39    toolConfig={"tools": tools},
40)
41
42print(response["output"])

Output:

 1{
 2  "message": {
 3    "role": "assistant",
 4    "content": [
 5      {
 6        "text": "Okay, let me check the current weather for New York:"
 7      },
 8      {
 9        "toolUse": {
10          "toolUseId": "tooluse_rRwaOoldTeiRiDZhTadP0A",
11          "name": "get_weather",
12          "input": {
13            "location": "New York, NY",
14            "unit": "fahrenheit"
15          }
16        }
17      }
18    ]
19  }
20}

The output includes a toolUse object that indicates to us, the developers, that the AI is using the get_weather tool to fetch the current weather in New York. We must now fulfill the tool request by responding with the weather information.

But first, let’s build a simplistic router that can handle the tool request and respond with the weather information.

1def get_weather(location: str, unit: str = "fahrenheit") -> dict:
2    return {"temperature": "78"}
3
4def tool_router(tool_name, input):
5    match tool_name:
6        case "get_weather":
7            return get_weather(input["location"], input.get("unit", "fahrenheit"))
8        case _:
9            raise ValueError(f"Unknown tool: {tool_name}")

Now, let’s update the code to handle the tool request and respond with the weather information.

 1import boto3
 2
 3def get_weather(location: str, unit: str = "fahrenheit") -> dict:
 4    return {"temperature": "78"}
 5
 6def tool_router(tool_name, input):
 7    match tool_name:
 8        case "get_weather":
 9            return get_weather(input["location"], input.get("unit", "fahrenheit"))
10        case _:
11            raise ValueError(f"Unknown tool: {tool_name}")
12
13client = boto3.client("bedrock-runtime")
14
15tools = [
16    {
17        "toolSpec": {
18            "name": "get_weather",
19            "description": "Get the current weather in a given location",
20            "inputSchema": {
21                "json": {
22                    "type": "object",
23                    "properties": {
24                        "location": {
25                            "type": "string",
26                            "description": "The city and state, e.g. San Francisco, CA",
27                        },
28                        "unit": {
29                            "type": "string",
30                            "enum": ["celsius", "fahrenheit"],
31                            "description": "The unit of temperature, either 'celsius' or 'fahrenheit'",
32                        },
33                    },
34                    "required": ["location"],
35                }
36            },
37        }
38    },
39]
40messages = [
41    {
42        "role": "user",
43        "content": [{"text": "What is the weather like right now in New York?"}],
44    }
45]
46response = client.converse(
47    modelId="anthropic.claude-3-haiku-20240307-v1:0",
48    messages=messages,
49    toolConfig={"tools": tools},
50)
51
52ai_message = response["output"]["message"]
53messages.append(ai_message)
54
55if response["stopReason"] == "tool_use":
56    contents = response["output"]["message"]["content"]
57    for c in contents:
58        if "toolUse" not in c:
59            continue
60
61        tool_use = c["toolUse"]
62        tool_id = tool_use["toolUseId"]
63        tool_name = tool_use["name"]
64        input = tool_use["input"]
65
66        tool_result = {"toolUseId": tool_id}
67        try:
68            output = tool_router(tool_name, input)
69            if isinstance(output, dict):
70                tool_result["content"] = [{"json": output}]
71            elif isinstance(output, str):
72                tool_result["content"] = [{"text": output}]
73            # Add more cases, such as images, if needed
74            else:
75                raise ValueError(f"Unsupported output type: {type(output)}")
76        except Exception as e:
77            tool_result["content"] = [{"text": f"An unknown error occurred: {str(e)}"}]
78            tool_result["status"] = "error"
79
80        message = {"role": "user", "content": [{"toolResult": tool_result}]}
81        messages.append(message)
82
83    response = client.converse(
84        modelId="anthropic.claude-3-haiku-20240307-v1:0",
85        messages=messages,
86        toolConfig={"tools": tools},
87    )
88
89print(response["output"])

Output:

 1{
 2  "message": {
 3    "role": "assistant",
 4    "content": [
 5      {
 6        "text": "According to the weather data, the current temperature in New York, NY is 78 degrees Fahrenheit."
 7      }
 8    ]
 9  }
10}

Great! We have successfully responded to the AI tool request and provided the weather information for New York.

Step 5 - Using multiple tools

For reference, I’m adapting the Anthropic AI Tool examples from their documentation to the Bedrock Converse API.

In the previous example, we only used one tool to fetch the weather information. However, we can use multiple tools in a single conversation.

Let’s add another tool to the conversation to fetch the current time and introduce a loop to handle multiple tool requests.

  1import boto3
  2
  3def get_weather(location: str, unit: str = "fahrenheit") -> dict:
  4    return {"temperature": "78"}
  5
  6def get_time(timezone: str) -> str:
  7    return "12:00PM"
  8
  9def tool_router(tool_name, input):
 10    match tool_name:
 11        case "get_weather":
 12            return get_weather(input["location"], input.get("unit", "fahrenheit"))
 13        case "get_time":
 14            return get_time(input["timezone"])
 15        case _:
 16            raise ValueError(f"Unknown tool: {tool_name}")
 17
 18client = boto3.client("bedrock-runtime")
 19
 20tools = [
 21    {
 22        "toolSpec": {
 23            "name": "get_weather",
 24            "description": "Get the current weather in a given location",
 25            "inputSchema": {
 26                "json": {
 27                    "type": "object",
 28                    "properties": {
 29                        "location": {
 30                            "type": "string",
 31                            "description": "The city and state, e.g. San Francisco, CA",
 32                        },
 33                        "unit": {
 34                            "type": "string",
 35                            "enum": ["celsius", "fahrenheit"],
 36                            "description": "The unit of temperature, either 'celsius' or 'fahrenheit'",
 37                        },
 38                    },
 39                    "required": ["location"],
 40                }
 41            },
 42        }
 43    },
 44    {
 45        "toolSpec": {
 46            "name": "get_time",
 47            "description": "Get the current time in a given timezone",
 48            "inputSchema": {
 49                "json": {
 50                    "type": "object",
 51                    "properties": {
 52                        "timezone": {
 53                            "type": "string",
 54                            "description": "The IANA time zone name, e.g. America/Los_Angeles",
 55                        }
 56                    },
 57                    "required": ["timezone"],
 58                }
 59            },
 60        }
 61    },
 62]
 63messages = [
 64    {
 65        "role": "user",
 66        "content": [
 67            {
 68                "text": "What is the weather like right now in New York and what time is it there?"
 69            }
 70        ],
 71    }
 72]
 73response = client.converse(
 74    modelId="anthropic.claude-3-haiku-20240307-v1:0",
 75    messages=messages,
 76    toolConfig={"tools": tools},
 77)
 78
 79ai_message = response["output"]["message"]
 80messages.append(ai_message)
 81
 82tool_use_count = 0
 83while response["stopReason"] == "tool_use":
 84    if response["stopReason"] == "tool_use":
 85        contents = response["output"]["message"]["content"]
 86        for c in contents:
 87            if "toolUse" not in c:
 88                continue
 89
 90            tool_use = c["toolUse"]
 91            tool_id = tool_use["toolUseId"]
 92            tool_name = tool_use["name"]
 93            input = tool_use["input"]
 94
 95            tool_result = {"toolUseId": tool_id}
 96            try:
 97                output = tool_router(tool_name, input)
 98                if isinstance(output, dict):
 99                    tool_result["content"] = [{"json": output}]
100                elif isinstance(output, str):
101                    tool_result["content"] = [{"text": output}]
102                # Add more cases such as images if needed
103                else:
104                    raise ValueError(f"Unsupported output type: {type(output)}")
105            except Exception as e:
106                tool_result["content"] = [
107                    {"text": f"An unknown error occurred: {str(e)}"}
108                ]
109                tool_result["status"] = "error"
110
111            message = {"role": "user", "content": [{"toolResult": tool_result}]}
112            messages.append(message)
113
114        response = client.converse(
115            modelId="anthropic.claude-3-haiku-20240307-v1:0",
116            messages=messages,
117            toolConfig={"tools": tools},
118        )
119        ai_message = response["output"]["message"]
120        messages.append(ai_message)
121        tool_use_count += 1
122
123print(tool_use_count)
124print(response["output"])

Output:

 1{
 2  "message": {
 3    "role": "assistant",
 4    "content": [
 5      {
 6        "text": "The current time in New York is 12:00 PM.\n\nSo in summary, the weather in New York right now is 78 degrees Celsius, and the time is 12:00 PM."
 7      }
 8    ]
 9  }
10}

Tool use count: 2

We have successfully responded to the AI’s tool requests and provided the weather and time information for New York.

Conclusion

In this post, we learned how to use the Amazon Bedrock Converse API to augment converations with AI models. Not only did we leverage text and images, but we also used tools to simulate fetching external data.

The Amazon Bedrock Converse API is a powerful tool that can be used to build conversational AI applications!

Multiple operations in a single DynamoDB update expression

Thomas Taylor — Sun, 28 Apr 2024 13:10:00 -0400

DynamoDB supports an UpdateItem operation that modifies an existing or creates a new item. The UpdateItem accepts an UpdateExpression that dictates which operations will occur on the specified item.

In this post, we’ll explore how to perform multiple operations including within a clause/keyword and with multiple clauses/keywords.

What is an update expression

An update expression supports four keywords:

SET (modify or add item attributes)
REMOVE (deleting attributes from an item)
ADD (updating numbers and sets)
DELETE (removing elements from a set)

The syntax for an update expression is as follows:

1update-expression ::=
2    [ SET action [, action] ... ]
3    [ REMOVE action [, action] ...]
4    [ ADD action [, action] ... ]
5    [ DELETE action [, action] ...]

As you can view, DynamoDB supports four main clauses that begin with one of the specified operations.

For a detailed explanation of update expressions, please refer to the AWS documentation.

For demonstration purposes, I’m going to create a DynamoDB table named books with a number hash key named id.

1aws dynamodb create-table \
2  --table-name books \
3  --attribute-definitions AttributeName=id,AttributeType=N \
4  --key-schema AttributeName=id,KeyType=HASH \
5  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

I’ll also add an item to the table:

 1{
 2  "id": {
 3    "N": "1"
 4  },
 5  "title": {
 6    "S": "Iliad"
 7  },
 8  "author": {
 9    "S": "Homer"
10  },
11  "genre": {
12    "S": "Epic Poetry"
13  },
14  "publication_year": {
15    "N": "1488"
16  }
17}

1aws dynamodb put-item \
2    --table-name books \
3    --item file://book.json

How to specify a single keyword in DynamoDB update expression

Let’s showcase a basic use-case of adding an element to an existing item:

1aws dynamodb update-item \
2    --table-name books \
3    --key '{"id":{"N":"1"}}' \
4    --update-expression "SET category = :c" \
5    --expression-attribute-values '{":c":{"S":"Poem"}}'

Now the item has a new category attribute:

1aws dynamodb get-item \
2    --table-name books \
3    --key '{"id":{"N":"1"}}' \
4    --projection-expression "category"

Output:

1{
2  "Item": {
3    "category": {
4      "S": "Poem"
5    }
6  }
7}

How to specify a single keyword with multiple actions in DynamoDB update expression

To showcase mutliple SET operations in a single DynamoDB update expression, we’ll use a comma (,) to separate them.

In the following example, we’re changing the category from poem -> poetry and adding a new element named genre.

1aws dynamodb update-item \
2    --table-name books \
3    --key '{"id":{"N":"1"}}' \
4    --update-expression "SET category = :c, genre = :g" \
5    --expression-attribute-values '{":c":{"S":"poetry"}, ":g":{"S":"Epic poetry"}}' \
6    --return-values UPDATED_NEW

Output:

 1{
 2  "Attributes": {
 3    "category": {
 4      "S": "poetry"
 5    },
 6    "genre": {
 7      "S": "Epic poetry"
 8    }
 9  }
10}

As you see, the update-expression includes a comma between the two operations:

SET category = :c, genre = :g

How to specify multiple keywords with multiple actios in DynamoDB update expression

Lastly, let’s showcase leveraging multiple keywords with their respective actions.

We’re going to perform the following:

Set a new string attribute named product_category with a value of poem
Remove the category attribute
Add 1 to existing checkout_total number or initialize it with the number value of 1
Add Trojan War to existing subjects set or initialize it with the first index being string value of Trojan War

1aws dynamodb update-item \
2    --table-name books \
3    --key '{"id":{"N":"1"}}' \
4    --update-expression "SET #pc = :pc REMOVE #c ADD #ct :inc, #s :s" \
5    --expression-attribute-names '{"#s":"subjects","#pc":"product_category","#c":"category","#ct":"checkout_total"}' \
6    --expression-attribute-values '{":s":{"SS":["Trojan War"]},":pc":{"S":"poem"},":inc":{"N":"1"}}' \
7    --return-values UPDATED_NEW

Output:

 1{
 2  "Attributes": {
 3    "checkout_total": {
 4      "N": "1"
 5    },
 6    "subjects": {
 7      "SS": [
 8        "Trojan War"
 9      ]
10    },
11    "product_category": {
12      "S": "poem"
13    }
14  }
15}

It’s important to note that because the ADD operation will continue incrementing and the SET data structure doesn’t allow duplicates, we can rerun the same operation and checkout_total will increase by 1.

For readability, let’s substitute the expression-attribute-names and expression-attribute-values back into the update-expression:

1SET product_category = poem REMOVE category ADD checkout_total 1, subjects ["Trojan War"]

To summarize, using mutliple clauses in a DynamoDB UpdateExpression is supported by simply separating the clauses by keywords.

If you want to perform multiple operations using the same clause, use a comma (,) to separate them.

Detecting and redacting PII using Amazon Bedrock

Thomas Taylor — Tue, 23 Apr 2024 00:25:00 -0400

Typically, AWS recommends leveraging an existing service offering such as Amazon Comprehend to detect and redact PII. However, this post explores an alternative solution using Amazon Bedrock.

This is possible using the Claude, Anthropic’s large langauge model, and their publicly available prompt library. In our case, we’ll leverage the PII purifier prompt that is maintained by their prompt engineers.

How to extract PII using Amazon Bedrock in Python

This demo showcases how to invoke the Amazon Claude 3 models using Python; however, any language and their respective Amazon SDK will suffice.

Install boto3

Firstly, let’s install the AWS Python SDK, boto3.

1pip install boto3

Instantiate a client

Ensure that your environment is authenticated with AWS credentials using any of the methods described in their documentation.

Instantiate the bedrock runtime client like so:

1import boto3
2
3bedrock_runtime = boto3.client("bedrock-runtime")

Invoke the Claude model

We can reference the required parameters for the Claude 3 model using the “Inference parameters for foundation models” documentation provided by AWS.

In Claude 3’s case, the Messages API will be used like so:

 1import boto3
 2import json
 3
 4bedrock_runtime = boto3.client("bedrock-runtime")
 5response = bedrock_runtime.invoke_model(
 6    body=json.dumps(
 7        {
 8            "anthropic_version": "bedrock-2023-05-31",
 9            "max_tokens": 1000,
10            "messages": [{"role": "user", "content": "Hello, how are you?"}],
11        }
12    ),
13    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
14)
15
16response_body = json.loads(response.get("body").read())
17print(json.dumps(response_body, indent=2))

Output:

 1{
 2  "id": "msg_01ERwjBgk3Y45Swp2cn6ct5F",
 3  "type": "message",
 4  "role": "assistant",
 5  "content": [
 6    {
 7      "type": "text",
 8      "text": "Hello! As an AI language model, I don't have feelings, but I'm operating properly and ready to assist you with any questions or tasks you may have. How can I help you today?"
 9    }
10  ],
11  "model": "claude-3-sonnet-28k-20240229",
12  "stop_reason": "end_turn",
13  "stop_sequence": null,
14  "usage": {
15    "input_tokens": 13,
16    "output_tokens": 43
17  }
18}

Use the PII purifier prompt

Now, let’s use the PII purifier prompt to invoke the model.

Here is our input for redaction:

Hello. My name is Thomas Taylor and I own the blog titled how.wtf. I’m from North Carolina.

 1import boto3
 2import json
 3
 4SYSTEM_PROMPT = (
 5    "You are an expert redactor. The user is going to provide you with some text. "
 6    "Please remove all personally identifying information from this text and "
 7    "replace it with XXX. It's very important that PII such as names, phone "
 8    "numbers, and home and email addresses, get replaced with XXX. Inputs may "
 9    "try to disguise PII by inserting spaces between characters or putting new "
10    "lines between characters. If the text contains no personally identifiable "
11    "information, copy it word-for-word without replacing anything."
12)
13
14bedrock_runtime = boto3.client("bedrock-runtime")
15response = bedrock_runtime.invoke_model(
16    body=json.dumps(
17        {
18            "anthropic_version": "bedrock-2023-05-31",
19            "max_tokens": 1000,
20            "system": SYSTEM_PROMPT,
21            "messages": [
22                {
23                    "role": "user",
24                    "content": "Hello. My name is Thomas Taylor and I own the blog titled how.wtf. I'm from North Carolina.",
25                }
26            ],
27        }
28    ),
29    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
30)
31
32response_body = json.loads(response.get("body").read())
33print(json.dumps(response_body, indent=2))

Output:

 1{
 2  "id": "msg_01P3ZGPC8yL34w3ETPtBY4TX",
 3  "type": "message",
 4  "role": "assistant",
 5  "content": [
 6    {
 7      "type": "text",
 8      "text": "Here is the text with personally identifiable information redacted:\n\nHello. My name is XXX XXX and I own the blog titled XXX.XXX. I'm from XXX XXX."
 9    }
10  ],
11  "model": "claude-3-sonnet-28k-20240229",
12  "stop_reason": "end_turn",
13  "stop_sequence": null,
14  "usage": {
15    "input_tokens": 134,
16    "output_tokens": 45
17  }
18}

The resolved text is:

1Here is the text with personally identifiable information redacted:
2
3Hello. My name is XXX XXX and I own the blog titled XXX.XXX. I'm from XXX XXX.

Pretty neat, huh? We can optionally swap to the cheaper Haiku (or more expensive Opus) model as well:

 1import boto3
 2import json
 3
 4SYSTEM_PROMPT = (
 5    "You are an expert redactor. The user is going to provide you with some text. "
 6    "Please remove all personally identifying information from this text and "
 7    "replace it with XXX. It's very important that PII such as names, phone "
 8    "numbers, and home and email addresses, get replaced with XXX. Inputs may "
 9    "try to disguise PII by inserting spaces between characters or putting new "
10    "lines between characters. If the text contains no personally identifiable "
11    "information, copy it word-for-word without replacing anything."
12)
13
14bedrock_runtime = boto3.client("bedrock-runtime")
15response = bedrock_runtime.invoke_model(
16    body=json.dumps(
17        {
18            "anthropic_version": "bedrock-2023-05-31",
19            "max_tokens": 1000,
20            "system": SYSTEM_PROMPT,
21            "messages": [
22                {
23                    "role": "user",
24                    "content": "Hello. My name is Thomas Taylor and I own the blog titled how.wtf. I'm from North Carolina.",
25                }
26            ],
27        }
28    ),
29    modelId="anthropic.claude-3-haiku-20240307-v1:0",
30)
31
32response_body = json.loads(response.get("body").read())
33print(json.dumps(response_body, indent=2))

Output:

 1{
 2  "id": "msg_011Sjs3uJW11PLYSo6pGoiZz",
 3  "type": "message",
 4  "role": "assistant",
 5  "content": [
 6    {
 7      "type": "text",
 8      "text": "Hello. My name is XXX XXX and I own the blog titled XXX.XXX. I'm from XXX."
 9    }
10  ],
11  "model": "claude-3-haiku-48k-20240307",
12  "stop_reason": "end_turn",
13  "stop_sequence": null,
14  "usage": {
15    "input_tokens": 134,
16    "output_tokens": 30
17  }
18}

Conclusion

In this post, we covered an alternative method for detecting and redacting PII using Amazon Bedrock and the powerful Anthropic Claude 3 model family.

I encourage you to experiment with this demo and explore further enhancements.

Anthropic Claude with tools using Python SDK

Thomas Taylor — Sun, 21 Apr 2024 01:35:00 -0400

On April 4th, 2024, Anthropic released official support for tool use through their API. This feature allows developers to define one or more tools that include parameters, descriptions, and schema definitions for Claude to process.

In this post, we’ll go over how to leverage these tools using the Python Anthropic SDK.

Claude with tools example in Python

Firstly, ensure that your Anthropic API key is available as an environment variable in your shell:

1export ANTHROPIC_API_KEY="sk-......"

Install the Anthropic SDK

Next, install the latest version of the Anthropic SDK using pip.

1pip install anthropic

Define the tools

Using the provided JSON schema example in the Anthropic documentation, let’s define a get_current_stock_price tool:

 1{
 2  "name": "get_current_stock_price",
 3  "description": "Retrieves the current stock price for a given ticker symbol. The ticker symbol must be a valid symbol for a publicly traded company on a major US stock exchange like NYSE or NASDAQ. The tool will return the latest trade price in USD. It should be used when the user asks about the current or most recent price of a specific stock. It will not provide any other information about the stock or company.",
 4  "input_schema": {
 5    "type": "object",
 6    "properties": {
 7      "ticker": {
 8        "type": "string",
 9        "description": "The stock ticker symbol, e.g. AAPL for Apple Inc."
10      }
11    },
12    "required": ["ticker"]
13  }
14}

In Python,

 1import anthropic
 2
 3tools = [
 4    {
 5        "name": "get_current_stock_price",
 6        "description": "Retrieves the current stock price for a given ticker symbol. The ticker symbol must be a valid symbol for a publicly traded company on a major US stock exchange like NYSE or NASDAQ. The tool will return the latest trade price in USD. It should be used when the user asks about the current or most recent price of a specific stock. It will not provide any other information about the stock or company.",
 7        "input_schema": {
 8            "type": "object",
 9            "properties": {
10                "symbol": {
11                    "type": "string",
12                    "description": "The stock ticker symbol, e.g. AAPL for Apple Inc."
13                }
14            },
15            "required": ["symbol"]
16        }
17    }
18]

Instantiate a client

To invoke Claude, we need to instantiate a client. The SDK uses the ANTHROPIC_API_KEY environment variable by default for authentication.

1import anthropic
2
3client = anthropic.Anthropic()

Invoke Claude with tools

Now, let’s create a new message using Claude and the tools:

 1import anthropic
 2
 3tools = [
 4    {
 5        "name": "get_current_stock_price",
 6        "description": "Retrieves the current stock price for a given ticker symbol. The ticker symbol must be a valid symbol for a publicly traded company on a major US stock exchange like NYSE or NASDAQ. The tool will return the latest trade price in USD. It should be used when the user asks about the current or most recent price of a specific stock. It will not provide any other information about the stock or company.",
 7        "input_schema": {
 8            "type": "object",
 9            "properties": {
10                "symbol": {
11                    "type": "string",
12                    "description": "The stock ticker symbol, e.g. AAPL for Apple Inc."
13                }
14            },
15            "required": ["symbol"]
16        }
17    }
18]
19
20client = anthropic.Anthropic()
21
22response = client.beta.tools.messages.create(
23    model="claude-3-haiku-20240307",
24    max_tokens=1024,
25    tools=tools,
26    messages=[
27        {
28            "role": "user",
29            "content": "What is the stock price of Apple?"
30        }
31    ]
32)
33
34print(response)

Output:

1ToolsBetaMessage(id='msg_01Bedxru94A4Pe1sHgWtymSJ', content=[ToolUseBlock(id='toolu_01CbGgyko9mdkKSDPw6LsTvV', input={'symbol': 'AAPL'}, name='get_current_stock_price', type='tool_use')], model='claude-3-haiku-20240307', role='assistant', stop_reason='tool_use', stop_sequence=None, type='message', usage=Usage(input_tokens=433, output_tokens=60))

The output includes a ToolUseBlock with the input for the tool. We now need to process the tool and return the result to Claude.

Building a tool router

Let’s build a mock function that returns 150.0 for the stock price:

1def get_current_stock_price(symbol):
2    # Fake implementation for demonstration purposes
3    return 150.0

And then a “router” function that accepts the tool name and input:

1def process_tool(tool_name, input):
2    match tool_name:
3        case "get_current_stock_price":
4            return get_current_stock_price(input["symbol"])
5        case _:
6            raise ValueError(f"Unknown tool: {tool_name}")

Responding with a tool result example

Anthropic provides a Jupyter notebook that showcases a customer service example using tools. We’ll adapt some of the functionality to our script:

 1while response.stop_reason == "tool_use":
 2    tool_use = next(block for block in response.content if block.type == "tool_use")
 3    tool_name = tool_use.name
 4    tool_input = tool_use.input
 5    tool_result = process_tool(tool_name, tool_input)
 6
 7    messages = [
 8        {"role": "user", "content": user_message},
 9        {"role": "assistant", "content": response.content},
10        {
11            "role": "user",
12            "content": [
13                {
14                    "type": "tool_result",
15                    "tool_use_id": tool_use.id,
16                    "content": str(tool_result),
17                }
18            ],
19        },
20    ]
21
22    response = client.beta.tools.messages.create(
23        model=model,
24        max_tokens=4096,
25        tools=tools,
26        messages=messages
27    )
28
29final_response = next(
30    (block.text for block in response.content if isinstance(block, TextBlock)),
31    None,
32)

The code does the following:

Loop while the response is tool_use
Extracts the current tool’s name and input
Processes the tool by calling the router function
Creates a new Claude message using the prior messages and the tool result
Retrieves the final response from Claude and returns the text

Putting it altogether

Here’s the resulting code:

 1import anthropic
 2from anthropic.types import TextBlock
 3
 4client = anthropic.Anthropic()
 5
 6
 7def talk(client, tools, model, user_message):
 8    response = client.beta.tools.messages.create(
 9        model=model,
10        max_tokens=1024,
11        tools=tools,
12        messages=[{"role": "user", "content": user_message}],
13    )
14
15    while response.stop_reason == "tool_use":
16        tool_use = next(block for block in response.content if block.type == "tool_use")
17        tool_name = tool_use.name
18        tool_input = tool_use.input
19        tool_result = process_tool(tool_name, tool_input)
20
21        messages = [
22            {"role": "user", "content": user_message},
23            {"role": "assistant", "content": response.content},
24            {
25                "role": "user",
26                "content": [
27                    {
28                        "type": "tool_result",
29                        "tool_use_id": tool_use.id,
30                        "content": str(tool_result),
31                    }
32                ],
33            },
34        ]
35
36        response = client.beta.tools.messages.create(
37            model=model, max_tokens=4096, tools=tools, messages=messages
38        )
39
40    final_response = next(
41        (block.text for block in response.content if isinstance(block, TextBlock)),
42        None,
43    )
44
45    return final_response
46
47
48def get_current_stock_price(symbol: str):
49    # Fake implementation for demonstration purposes
50    return 150.0
51
52
53def process_tool(tool_name: str, input):
54    match tool_name:
55        case "get_current_stock_price":
56            return get_current_stock_price(input["symbol"])
57        case _:
58            raise ValueError(f"Unknown tool: {tool_name}")
59
60
61if __name__ == "__main__":
62    client = anthropic.Anthropic()
63    tools = [
64        {
65            "name": "get_current_stock_price",
66            "description": "Retrieves the current stock price for a given ticker symbol. The ticker symbol must be a valid symbol for a publicly traded company on a major US stock exchange like NYSE or NASDAQ. The tool will return the latest trade price in USD. It should be used when the user asks about the current or most recent price of a specific stock. It will not provide any other information about the stock or company.",
67            "input_schema": {
68                "type": "object",
69                "properties": {
70                    "symbol": {
71                        "type": "string",
72                        "description": "The stock ticker symbol, e.g. AAPL for Apple Inc.",
73                    }
74                },
75                "required": ["symbol"],
76            },
77        }
78    ]
79    response = talk(
80        client, tools, "claude-3-haiku-20240307", "What is the price of Apple?"
81    )
82    print(response)

Here is the output:

1The current stock price for Apple (ticker symbol AAPL) is $150.00.

Conclusion

In this post, we covered:

How to invoke Claude using tools
How to process tool calls using a switch statement
How to respond with a tool_result so that Claude can output a final answer

The purpose of this post was to get you started by providing a foundational understanding of Anthropic tool usage using the Python SDK. For more information about tool best practices or multi-tool chaining, please refer to Anthropic’s documentation.

Applying event filters to AWS Lambda Functions with the AWS CDK

Thomas Taylor — Sun, 14 Apr 2024 12:30:00 -0400

The primary motive for writing this article is to address the common error I repeatedly received while troubleshooting event filters:

Invalid filter pattern definition. (Service: AWSLambda; Status Code: 400; Error Code: InvalidParameterValueException

For my case, the goal was to invoke an AWS Lambda function via a DynamoDB stream.

What are Lambda Event Filters

Lambda event filters allow developers to specify which types of records from a stream or queue are submitted to a Lambda. Event filters are included in event source mapping definitions so that Lambda functions are only invoked when the filter criteria is met.

DynamoDB TTL deletion event filter

My use case involved invoking a lambda function to archive resources when a DynamoDB TTL expiry was met. Fortunately, the DynamoDB documentation has a section that describes how to achieve this.

The required filter criteria is as follows:

 1{
 2  "Filters": [
 3    {
 4      "Pattern": {
 5        "userIdentity": {
 6          "type": ["Service"],
 7          "principalId": ["dynamodb.amazonaws.com"]
 8        }
 9      }
10    }
11  ]
12}

This filter patterns suggests that only actions submitted by the service principal dynamodb.amazonaws.com should be processed by the receiving consumer. This makes sense because the DynamoDB service deletes expired TTL items on our behalf.

Adding event filters to Lambda Functions with AWS CDK

The following example demonstrates how to add event filters to a Lambda function using the AWS CDK in TypeScript:

 1import * as path from 'path';
 2
 3import { 
 4  Code, 
 5  FilterCriteria,
 6  Function,
 7  Runtime,
 8  StartingPosition
 9} from 'aws-cdk-lib/aws-lambda';
10
11import { AttributeType, StreamViewType, Table } from 'aws-cdk-lib/aws-dynamodb';
12
13import { DynamoEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';
14
15const table = new Table(this, 'Table', {
16  partitionKey: {
17    name: 'id',
18    type: AttributeType.STRING
19  },
20  stream: StreamViewType.NEW_IMAGE
21});
22
23const lambda = new Function(this, 'Function', {
24  runtime: Runtime.NODEJS_20_X,
25  handler: 'index.handler',
26  code: Code.fromAsset(path.join(__dirname, 'lambda-handler'))
27});
28
29lambda.addEventSource(
30  new DynamoEventSource(databaseTable, {
31    startingPosition: StartingPosition.TRIM_HORIZON,
32    filters: [
33      FilterCriteria.filter({
34        userIdentity: {
35          type: ['Service'],
36          principalId: ['dynamodb.amazonaws.com']
37        }
38      })
39    ]
40  })
41)

The crucial part is the inner filters attribute within the event source definition:

 1lambda.addEventSource(
 2  new DynamoEventSource(databaseTable, {
 3    startingPosition: StartingPosition.TRIM_HORIZON,
 4    filters: [
 5      FilterCriteria.filter({
 6        userIdentity: {
 7          type: ['Service'],
 8          principalId: ['dynamodb.amazonaws.com']
 9        }
10      })
11    ]
12  })
13)

It’s important to note that the static method of FilterCriteria.filter adds the pattern top-level attribute and marshals the inner JSON on our behalf.

As of April 2024, the filters attribute is available on many supported event sources:

The DynamoEventSource documentation definition
The KafkaEventSource documentation definition
The KinesisEventSource documentation definition
The ManagedKafkaEventSource documentation definition
The S3EventSource documentation definition
The SelfManagedKafkaEventSourceProps documentation definition
The SqsEventSourceProps documentation definition

Deploy serverless Lambda TypeScript API with function url using AWS CDK

Thomas Taylor — Sat, 06 Apr 2024 16:20:00 -0400

In November 2023, I wrote a post describing how to deploy a lambda function with a function url in Python. For this post, I want to showcase how streamlined and practical it is to deploy a “Lambdalith” (a single Lambda function) that contains an entire API.

What this means:

No API Gateway
API requests can take longer than 30 seconds
Faster deployments
Local testing without cloud deployment
Reduced costs *
Easy management *

= Depends on the use-case

How to deploy a serverless API using Fastify

To begin, let’s initialize a CDK application for Typescript:

cdk init app --language typescript

This creates the boilerplate directories and files we’ll need:

 1serverless-api
 2├── README.md
 3├── bin
 4│   └── serverless-api.ts
 5├── cdk.json
 6├── jest.config.js
 7├── lib
 8│   └── serverless-api-stack.ts
 9├── package-lock.json
10├── package.json
11├── test
12│   └── serverless-api.test.ts
13└── tsconfig.json

Install and configure Fastify

Fastify is a JavaScript web framework that intentionally aims for low overhead and speed over other frameworks such as express. I have arbitrarily chose it for this tutorial, but any web framework that supports routing will work.

Install fastify

Install fastify using one of the methods described in their documentation and their AWS Lambda adapter @fastify/aws-lambda.

For this tutorial, I’ll use npm.

npm i fastify @fastify/aws-lambda @types/aws-lambda

Create an entry file

To make it easy, we’ll create an entry point for the lambda at handler/index.ts with the following contents:

 1import Fastify from "fastify";
 2import awsLambdaFastify from "@fastify/aws-lambda";
 3
 4function init() {
 5  const app = Fastify();
 6  app.get('/', (request, reply) => reply.send({ hello: 'world' }));
 7  return app;
 8}
 9
10if (require.main === module) {
11  // called directly i.e. "node app"
12  init().listen({ port: 3000 }, (err) => {
13    if (err) console.error(err);
14    console.log('server listening on 3000');
15  });
16} else {
17  // required as a module => executed on aws lambda
18  exports.handler = awsLambdaFastify(init())
19}

The directory structure should look like the following tree:

 1serverless-api
 2├── README.md
 3├── bin
 4│   └── serverless-api.ts
 5├── cdk.json
 6├── handler
 7│   └── index.ts
 8├── jest.config.js
 9├── lib
10│   └── serverless-api-stack.ts
11├── package-lock.json
12├── package.json
13├── test
14│   └── serverless-api.test.ts
15└── tsconfig.json

With this method, we are able to test locally without deploying to the cloud.

First, transpile the Typescript files to JavaScript:

1npm run build

Then execute the handler/index.js file with node:

1node handler/index.js

If you visit http://localhost:3000 in your browser, it should display:

1{
2    "hello": "world"
3}

Deploying the function with the function url enabled

Fortunately, the AWS CDK enables users to quickly deploy using the NodeJSFunction construct. Replace the contents of serverless-api-stack.ts with the following snippet:

 1import * as cdk from 'aws-cdk-lib';
 2import { Construct } from 'constructs';
 3import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';
 4import { FunctionUrlAuthType } from 'aws-cdk-lib/aws-lambda';
 5
 6export class ServerlessApiStack extends cdk.Stack {
 7  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
 8    super(scope, id, props);
 9    const handler = new NodejsFunction(this, 'handler', {
10      entry: './handler/index.ts',
11      timeout: cdk.Duration.minutes(1)
12    });
13    const url = handler.addFunctionUrl({
14      authType: FunctionUrlAuthType.NONE
15    });
16    new cdk.CfnOutput(this, 'url', {
17      value: url.url
18    });
19  }
20}

The code creates a NodejsFunction lambda, enables the function url without authentication, and outputs the url as a CloudFormation export.

Deploy the stack using the cdk:

1npx cdk deploy

The command output contains the CfnOutput value:

Do you wish to deploy these changes (y/n)? y
ServerlessApiStack: deploying... [1/1]
ServerlessApiStack: creating CloudFormation changeset...

 ✅  ServerlessApiStack

✨  Deployment time: 38.19s

Outputs:
ServerlessApiStack.url = https://{id}.lambda-url.us-east-1.on.aws/
Stack ARN:
arn:aws:cloudformation:us-east-1:123456789012:stack/ServerlessApiStack/{id}

If you navigate to the url, you will view the expected results displayed:

1{
2    "hello": "world"
3}

All of this was completed with very little infrastructure to manage and a single index.ts file. From here, you can expand the project to include as many routes as you prefer.

Using Claude 3 Opus with Anthropic API in Python

Thomas Taylor — Sat, 09 Mar 2024 12:45:00 -0500

With the recent release of the Claude 3 family, this was the perfect opportunity to use the Claude API access that I was recently granted.

The Claude 3 family boasts a huge improvement from the prior family, Claude 2. The introductory post goes into greater detail.

In this post, we’ll explore how to invoke Claude 3 Opus using the Anthropic SDK.

Getting started

For the purposes of this post, we’ll leverage the Python Anthropic SDK.

1pip3 install anthropic

To authorize requests, please export the ANTHROPIC_API_KEY:

1export ANTHROPIC_API_KEY="sk..."

How to invoke Claude 3 Opus

The API for invoking Anthropic models is simple.

The model id we’ll use is the Opus release from Feb. 29th, 2024: claude-3-opus-20240229.

If you wish to target other models, please refer to Anthropic’s model page for more information.

 1from anthropic import Anthropic
 2
 3model = "claude-3-opus-20240229"
 4
 5client = Anthropic()
 6
 7message = client.messages.create(
 8    max_tokens=1024,
 9    messages=[{"role": "user", "content": "Hello! How are you?"}],
10    model=model,
11)
12print(message.content)
13print(f"Input tokens: {message.usage.input_tokens}")
14print(f"Output tokens: {message.usage.output_tokens}")

Output:

1[ContentBlock(text="Hello! I'm doing well, thank you for asking. As an AI language model, I don't have feelings, but I'm functioning properly and ready to assist you with any questions or tasks you may have. How can I help you today?", type='text')]
2Input tokens 13
3Output tokens 53

In the example above, the Anthropic() client was instantiated and automatically referenced the environment variable from earlier.

Then, using the Messages API, we created a new message.

How to invoke Claude 3 Opus with Streaming

Similarly to before, we also have the option to stream output using Anthropic’s API.

 1from anthropic import Anthropic
 2
 3model = "claude-3-opus-20240229"
 4
 5client = Anthropic()
 6
 7stream = client.messages.create(
 8    max_tokens=1024,
 9    messages=[
10        {
11            "role": "user",
12            "content": "Hello! How are you?",
13        }
14    ],
15    model=model,
16    stream=True,
17)
18
19message = ""
20input_tokens = None
21output_tokens = None
22for event in stream:
23    match event.type:
24        case "message_start":
25            input_tokens = event.message.usage.input_tokens
26        case "content_block_start":
27            message += event.content_block.text
28        case "content_block_delta":
29            message += event.delta.text
30        case "message_delta":
31            output_tokens = event.usage.output_tokens
32        case "content_block_stop" | "message_stop":
33            ...
34
35print(message)
36print(f"Input tokens: {input_tokens}")
37print(f"Output tokens: {output_tokens}")

Output:

1Hello! As an AI language model, I don't have feelings, but I'm functioning well and ready to assist you. How can I help you today?
2Input tokens: 13
3Output tokens: 35

For the sake of the example, I concatenated the “text”-based events together to showcase the full output.

For asynchronous eventing, please refer to the Anthropic SDK documentation.

Amazon Bedrock with images using Claude 3 Sonnet with Python

Thomas Taylor — Mon, 04 Mar 2024 23:30:00 -0500

On March 4th, 2024, the Claude 3 Sonnet foundational model was made available in Amazon Bedrock. Not only is the model a major improvement over the Claude 2.0 family, but it allows for vision input.

In the post, we’ll explore how to invoke Claude 3 Sonnet using Amazon’s boto3 library in Python.

Getting started

For those unfamiliar with the Bedrock runtime API, I have a post that demonstrates how to invoke it using Claude 2.0 and Python boto3. For this post, I’ll be focusing specifically on image invocations.

Firstly, install boto3

1pip3 install boto3

How to send images to Amazon Bedrock

Fortunately, the API is straight forward to use. Let’s instantiate the Amazon Bedrock Runtime client:

1import boto3
2
3runtime = boto3.client("bedrock-runtime")

For Claude 3 Sonnet, we are required to use the Anthropic Claude Messages API as the inference parameter input.

 1{
 2    "anthropic_version": "bedrock-2023-05-31", 
 3    "max_tokens": 1024,
 4    "messages": [
 5        {
 6            "role": "user",
 7            "content": [
 8                {
 9                    "type": "image",
10                    "source": {
11                        "type": "base64",
12                        "media_type": "image/jpeg",
13                        "data": "..."
14                    }
15                },
16                {
17                    "type": "text",
18                    "text": "What is in this image?"
19                }
20            ]
21        }
22    ]
23}

For the Claude 3 Sonnet model, the image is supplied using base64.

For my case, I will be using a local file for the model.

1with open("image.jpg", "rb") as image_file:
2    image_bytes = image_file.read()
3
4encoded_image = base64.b64encode(image_bytes).decode("utf-8")

You may choose to download image using the requests library.

1image_url = "https://url-to-the-image.com/file.jpg"
2image_bytes = requests.get(url).content
3
4encoded_image = base64.b64encode(image_bytes).decode("utf-8")

The image I’m using is the Hohenzollern Castle in Germany.

Here is the full example:

 1import base64
 2import json
 3
 4import boto3
 5
 6runtime = boto3.client("bedrock-runtime")
 7
 8with open("image.jpg", "rb") as image_file:
 9    image_bytes = image_file.read()
10
11encoded_image = base64.b64encode(image_bytes).decode("utf-8")
12
13body = json.dumps(
14    {
15        "anthropic_version": "bedrock-2023-05-31",
16        "max_tokens": 1000,
17        "messages": [
18            {
19                "role": "user",
20                "content": [
21                    {
22                        "type": "image",
23                        "source": {
24                            "type": "base64",
25                            "media_type": "image/jpeg",
26                            "data": encoded_image,
27                        },
28                    },
29                    {"type": "text", "text": "What is in this image?"},
30                ],
31            }
32        ],
33    }
34)
35
36response = runtime.invoke_model(
37    modelId="anthropic.claude-3-sonnet-20240229-v1:0",
38    body=body
39)
40
41response_body = json.loads(response.get("body").read())
42
43print(response_body)

Here’s the response_body converted to a JSON payload:

 1{
 2  "id":"msg_01315rokf5kw6LSnnW6aWyVb",
 3  "type":"message",
 4  "role":"assistant",
 5  "content":[
 6    {
 7      "type":"text",
 8      "text":"This image shows the famous Hohenzollern Castle in Germany. The castle sits majestically atop a cliff, overlooking a river or lake in the foreground. The massive medieval fortress features ornate towers, spires, and archways in a Romanesque architectural style with red tiled roofs. The castle appears to be well-preserved and maintained, set against a picturesque blue sky with fluffy white clouds. The surrounding scenery includes bare trees and a small playground area near the water's edge, suggesting this is a popular tourist destination showcasing Germany's rich historical heritage and stunning natural landscapes."
 9    }
10  ],
11  "model":"claude-3-sonnet-28k-20240229",
12  "stop_reason":"end_turn",
13  "stop_sequence":null,
14  "usage":{
15    "input_tokens":1606,
16    "output_tokens":130
17  }
18}

Sure enough, the model accurately depicted the image shown.

Create and run Bash programs with LangChain

Thomas Taylor — Sun, 11 Feb 2024 11:45:00 -0500

Using LangChain, we can create a prompt to output a bash, which is then executed via the BashProcess tool.

All of this is as easy as a few lines of Python code!

How to generate bash scripts using LangChain

First, let’s install the necessary dependencies:

1pip install langchain-core langchain-experimental langchain-openai

Now, let’s write a simplistic prompt to get OpenAI to output bash code.

For this example, we’ll target a script that outputs:

1Hello world!

Script:

 1from langchain_core.output_parsers import StrOutputParser
 2from langchain_experimental.llm_bash.bash import BashProcess
 3from langchain_openai import ChatOpenAI
 4from langchain_core.prompts import ChatPromptTemplate
 5
 6
 7def sanitize_output(text: str):
 8    _, after = text.split("```bash")
 9    return after.split("```")[0]
10
11
12template = """Write some bash code to solve the user's problem. 
13
14Return only bash code in Markdown format, e.g.:
15
16```bash
17....
18```"""
19prompt = ChatPromptTemplate.from_messages([("system", template), ("human", "{input}")])
20
21model = ChatOpenAI()
22
23chain = prompt | model | StrOutputParser() | sanitize_output | BashProcess().run
24
25print(chain.invoke({"input": "Print 'Hello world!' to the console"}))

Output:

1Hello world!

Using LangChain Expression Language (LCEL), we constructed a chain by using the pipe operator to combine:

prompt
model
StrOutputParser
sanitize_output
BashProcess().run

This example was adapted from the “Code Writing” cookbook provided in the Langchain documentation.

The sanitize_output function extracts the bash code from the markdown output and pipes it to the BashProcess tool. To learn more about the bash process tool, visit the documentation.

Step by step guide for the date in Bash

Thomas Taylor — Sat, 10 Feb 2024 22:00:07 -0500

Unix provides a utility named date. As its name implies, it allows you to fetch your system’s date.

Display the current date

Displaying the current date is simple:

1echo $(date)

Output:

1Sat Feb 10 21:29:37 EST 2024

The system date was printed with the timezone information.

Display the current date in UTC

Similarly, we can display the current date in UTC (Coordinated Universal Time) using the -u flag:

1echo $(date -u)

Output:

1Sun Feb 11 02:31:06 UTC 2024

The datetime was printed with the UTC timezone information.

Format the current date

The default date output can easily be formatted in a bash script:

1now="$(date '+%Y-%m-%d')"
2echo $now

Output:

12024-02-10

The date manual page showcases all the formatting options available.

Getting a date from days ago

Using the -d or --date option, we can display a different date:

1now="$(date -d '2 days ago' '+%Y-%m-%d')"
2echo $now

Output:

12024-02-08

For MacOS users, use the -v option:

1now="$(date -v-2d '+%Y-%m-%d')"
2echo $now

ISO-8601 date in Bash

In 2011, the -I or --iso-8601 option was reintroduced to the date utility.

1now="$(date -Iseconds)"
2echo $now

Output:

12024-02-11T02:47:41+00:00

For MacOS users, the -I option does not exist. You must format instead:

1now="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
2echo $now

Output:

12024-02-11T02:54:53Z

Comparing two dates in Bash

Using -d to convert the dates to unix timestamp, we can compare them.

1dateOne=$(date -d 2022-02-10 +%s)
2dateTwo=$(date -d 2021-12-12 +%s)
3
4if [ $dateOne -ge $dateTwo ]; then
5    echo "woah!"
6fi

For MacOS users, the equivalent option is:

1dateOne=$(date -j -f "%F" 2022-02-10 +"%s")
2dateTwo=$(date -j -f "%F" 2021-12-12 +"%s")
3
4if [ $dateOne -ge $dateTwo ]; then
5    echo "woah!"
6fi

Get Anthropic Claude Amazon Bedrock token counts

Thomas Taylor — Sat, 10 Feb 2024 01:10:00 -0500

Last year, I wrote a post showcasing how to tokenize words using the Anthropic SDK to track token usage.

While this method continues to work, it’s not necessary since the Amazon Bedrock team added token metrics to the SDK response metadata.

In this guide, we’ll explore the SDK and how to get those token counts.

Install the AWS SDK

To begin, install boto3, the AWS SDK for Python, using pip:

1pip install boto3

List model access

The invoke_model call requires a model id. To determine which models are accessible in your AWS account, run the following command using the aws cli:

1aws bedrock list-foundation-models \
2    --by-provider anthropic \
3    --query "modelSummaries[*].modelId"

Output:

 1[
 2    "anthropic.claude-instant-v1:2:100k",
 3    "anthropic.claude-instant-v1",
 4    "anthropic.claude-v1",
 5    "anthropic.claude-v2:0:18k",
 6    "anthropic.claude-v2:0:100k",
 7    "anthropic.claude-v2:1:18k",
 8    "anthropic.claude-v2:1:200k",
 9    "anthropic.claude-v2:1",
10    "anthropic.claude-v2"
11]

Call Claude using Bedrock client

The AWS SDK provides a bedrock-runtime client with an invoke_model function. We’ll use this and the Claude inference parameters documentation to invoke it.

 1import boto3
 2import json
 3
 4bedrock = boto3.client("bedrock-runtime")
 5
 6params = {
 7    "prompt": "\n\nHuman: Who are you?\n\nAssistant:",
 8    "max_tokens_to_sample": 200,
 9}
10
11response = bedrock.invoke_model(
12    body=json.dumps(params).encode(),
13    modelId="anthropic.claude-instant-v1"
14)
15
16print(response)

Here’s the output pretty-printed in json format:

 1{
 2  "ResponseMetadata": {
 3    "RequestId": "d72967be-90c2-4118-951e-a555455d5d7a",
 4    "HTTPStatusCode": 200,
 5    "HTTPHeaders": {
 6      "date": "Sat, 10 Feb 2024 05:55:23 GMT",
 7      "content-type": "application/json",
 8      "content-length": "117",
 9      "connection": "keep-alive",
10      "x-amzn-requestid": "d72967be-90c2-4118-951e-a555455d5d7a",
11      "x-amzn-bedrock-invocation-latency": "430",
12      "x-amzn-bedrock-output-token-count": "17",
13      "x-amzn-bedrock-input-token-count": "13"
14    },
15    "RetryAttempts": 0
16  },
17  "contentType": "application/json",
18  "body": "..."
19}

Notice the information it provides us in the ResponseMetadata.HTTPHeaders?

Count the Anthropic Claude token input and output

We can modify the code snippet above to reveal the model’s output and token usage:

 1import boto3
 2import json
 3
 4bedrock = boto3.client("bedrock-runtime")
 5
 6params = {
 7    "prompt": "\n\nHuman: Who are you?\n\nAssistant:",
 8    "max_tokens_to_sample": 200,
 9}
10
11response = bedrock.invoke_model(
12    body=json.dumps(params).encode(), modelId="anthropic.claude-instant-v1"
13)
14
15body = response["body"].read()
16data = json.loads(body)
17completion = data.get("completion")
18
19input_token_count = response["ResponseMetadata"]["HTTPHeaders"][
20    "x-amzn-bedrock-input-token-count"
21]
22
23output_token_count = response["ResponseMetadata"]["HTTPHeaders"][
24    "x-amzn-bedrock-output-token-count"
25]
26
27print(completion, input_token_count, output_token_count, sep="\n")

Output:

1 My name is Claude. I'm an AI assistant created by Anthropic.
213
320

That was easy!

The right way to test exit codes in Bash

Thomas Taylor — Sat, 03 Feb 2024 22:45:00 -0500

In this article, we’ll go over how to properly test exit codes in Bash scripts.

Testing exit codes with $?

When I first started writing Bash, I often wrote the following code to test whether a program’s execution was successful:

1some_command
2if [ "$?" -eq 0 ]; then
3    echo "It worked!"
4fi

However, this introduces complexity in the flow of the script. For example, one minor mistake like adding en echo statement:

1some_command
2echo "some_command"
3if [ "$?" -eq 0 ]; then
4    echo "It worked!"
5fi

can introduce unintended side effects that are not immediately clear. In this case, the echo statement’s exit status is 0 and skips the if conditional.

Therefore, if status codes are used as true/false indictators, do not use this method

Testing for successful exit codes

For the majority of my use cases, I need to evaluate whether a program’s execution is or is not successful.

Using the following syntax clearly informs readers what is happening:

1if some_command; then
2    echo "It worked!"
3fi

If some_command executes correctly, echo "It worked!".

Testing for failed exit codes

Similarly, we can test for failed exit codes by negating the if statement:

1if ! some_command; then
2    echo "It did not work!"
3fi

If some_command executes incorrectly, echo "It did not work!"

Testing for specific exit codes

If your use case requires testing for specific exit codes, then I recommend collecting the exit status on the same line and using a case statement.

For example:

1some_command; some_command_exit_code=$?
2case $some_command_exit_code in
3    0) echo "It worked!" ;;
4    *) echo "It did not work!" ;;
5esac

Using a case statement allows us to control the flow in an easy-to-digest manner.

After all, code maintainability should be prioritized.

How to use Embedchain with AWS Bedrock

Thomas Taylor — Sat, 03 Feb 2024 11:50:00 -0500

LangChain is a framework that streamlines developing LLM applications. In a nutshell, it provides abstractions that help build out common steps and flows for working with LLMs.

In this post, we’ll go over how to integrate Embedchain, an open source RAG framework, with AWS Bedrock!

What is Embedchain

Embedchain is an open source RAG framework aimed at allowing developers to quickly gather necessary knowledge for LLM context when answering user prompts.

In particular, it shines with its easy-to-use API for adding unstructured data into manageable and optimized chunks for retrieval. In addition, it has synchronization options to ensure data sources are frequently updated.

Getting started with Embedchain

To showcase the power of Embedchain, let’s spin up a quick demo about Ada Lovelace - “The world’s first programmer”.

To begin, install the necessary dependencies:

1pip install embedchain

For the sake of this example, I’m using OpenAI’s API. Embedchain supports many other models as well.

1export OPENAI_API_KEY="..."

Embedchain example RAG

The following code is a basic example of RAG using Embedchain.

1from embedchain import App
2
3bot = App()
4
5bot.add("https://en.wikipedia.org/wiki/Ada_Lovelace")
6
7print(bot.query("Who is Ada Lovelace?"))

Output:

1Inserting batches in chromadb: 100% 1/1 [00:00<00:00,  1.26it/s]
2Successfully saved https://en.wikipedia.org/wiki/Ada_Lovelace (DataType.WEB_PAGE). New chunks count: 33
3Ada Lovelace was an English mathematician and writer known for her work on Charles Babbage's proposed mechanical general-purpose computer, the Analytical Engine. She was the first to recognize that the machine had applications beyond pure calculation. Ada Lovelace was the only legitimate child of poet Lord Byron and reformer Lady Byron. She made significant contributions to the field of mathematics and is considered to be the world's first computer programmer.

Steps:

Instantiate a Pipeline from embedchain
Insert new documents into the pipeline
Automatically chunk and batch the documents into chromadb
Query the chromadb store for relevant documents and answer the question using OpenAI’s API

For more information on ChromaDB as a vector store, checkout my guide.

Using Embedchain with AWS Bedrock and Claude

Fortunately, Embedchain directly supports AWS Bedrock as an LLM.

Install the dependencies

1pip install embedchain[aws-bedrock]

We must install the aws-bedrock extension for embedchain.

Create a config.yaml file

Add the following config.yaml file:

1llm:
2  provider: aws_bedrock
3  config:
4    model: anthropic.claude-v2
5    model_kwargs:
6      temperature: 0.5
7      top_p: 1
8      top_k: 250
9      max_tokens_to_sample: 200

The model_kwargs in this configuration are the defaults from the model’s arguments specified in the AWS documentation.

Create main.py file

Before running the code, ensure you have AWS access. In my case, I’m using an AWS profile, but you can use any of the supported boto3 authentication methods.

1from embedchain import App
2
3bot = App.from_config(config_path="config.yaml")
4
5bot.add("https://en.wikipedia.org/wiki/Ada_Lovelace")
6
7print(bot.query("Who is Ada Lovelace?"))

Output:

 1Ada Lovelace was an English mathematician and writer. Some key facts about her:
 2
 3- She was the only legitimate child of poet Lord Byron and his wife Lady Byron.
 4
 5- She is chiefly known for her work on Charles Babbage's proposed mechanical general-purpose computer, the Analytical Engine. She recognized the machine had applications beyond pure calculation and published the first algorithm intended to be carried out by such a machine.
 6
 7- She was the first to recognize the full potential of a "computing machine" and of "programming". For this reason she is often considered the first computer programmer.
 8
 9- She was interested in mathematics and logic from an early age, and pursued studies assiduously despite often being ill in her childhood.
10
11- She married William King in 1835. He was made Earl of Lovelace in 1838, making her Countess of Lovelace.
12
13- She died of uterine cancer in 1852 at age

How to use JSON files in vector stores with Langchain

Thomas Taylor — Sun, 28 Jan 2024 05:35:00 -0500

Retrieval-Augmented Generation (RAG) is a technique for including contextual information from external sources in a large language model’s (LLM) prompt. In other terms, RAG is used to supplement an LLM’s answer by providing more information in the prompt; thus, removing the need to retrain or fine-tune a model.

For the purposes of this post, we will implement RAG by using Chroma DB as a vector store with the Nobel Prize data set.

Langchain with JSON data in a vector store

Chroma DB will be the vector storage system for this post. It’s easy to use, open-source, and provides additional filtering options for associated metadata.

Getting started

To begin, install langchain, langchain-community, chromadb and jq.

1pip install langchain langchain-community chromadb jq

jq is required for the JSONLoader class. Its purpose is to parse the JSON file and its contents.

Fetching relevant documents using document loaders

For reference, the prize.json file has the following schema:

 1{
 2    "prizes": [
 3        {
 4            "year": "string",
 5            "category": "string",
 6            "laureates": [
 7                {
 8                    "id": "string",
 9                    "firstname": "string",
10                    "surname": "string",
11                    "motivation": "string",
12                    "share": "string"
13                }
14            ]
15        }
16    ]
17}

with the laureates being optional since some years do not have any winners.

The following code is used to simply fetch relevant documents from Chroma.

 1from langchain_community.document_loaders import JSONLoader
 2from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
 3from langchain_community.vectorstores import Chroma
 4
 5embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
 6
 7loader = JSONLoader(file_path="./prize.json", jq_schema=".prizes[]", text_content=False)
 8documents = loader.load()
 9
10db = Chroma.from_documents(documents, embedding_function)
11query = "What year did albert einstein win the nobel prize?"
12docs = db.similarity_search(query)
13print(docs[0].page_content)

Steps:

Use the SentenceTransformerEmbeddings to create an embedding function using the open source model of all-MiniLM-L6-v2 from huggingface.
Instantiate the loader for the JSON file using the ./prize.json path.
Load the files
Instantiate a Chroma DB instance from the documents & the embedding model
Perform a cosine similarity search
Print out the contents of the first retrieved document

Output:

1{"year": "1921", "category": "physics", "laureates": [{"id": "26", "firstname": "Albert", "surname": "Einstein", "motivation": "\"for his services to Theoretical Physics, and especially for his discovery of the law of the photoelectric effect\"", "share": "1"}]}

The text_content=False flag allows each array JSON document entry to be vectorized. This works for this use case, but you may want something different.

For example, let’s vectorize the contents of the motivation text attribute and add metadata about each motivation:

 1from langchain_community.document_loaders import JSONLoader
 2from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
 3from langchain_community.vectorstores import Chroma
 4
 5
 6def metadata_func(record: dict, metadata: dict) -> dict:
 7    metadata["id"] = record.get("id")
 8    metadata["firstname"] = record.get("firstname")
 9    metadata["surname"] = record.get("surname")
10    metadata["share"] = record.get("share")
11    return metadata
12
13
14embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
15
16loader = JSONLoader(
17    file_path="./prize.json",
18    jq_schema=".prizes[].laureates[]?",
19    content_key="motivation",
20    metadata_func=metadata_func,
21)
22documents = loader.load()
23print(documents)

Steps:

Use the SentenceTransformerEmbeddings to create an embedding function using the open source model of all-MiniLM-L6-v2 from huggingface.
Instantiate the loader for the JSON file using the ./prize.json path.
Use the ? jq syntax to ignore nullables if laureates does not exist on the entry
Use a metadata_func to grab the fields of the JSON to put in the document’s metadata
Use the content_key to specify which field is used for the vector text
Load the files
Instantiate a Chroma DB instance from the documents & the embedding model
Print out the loaded documents

Output:

 1[
 2    Document(
 3        page_content='"for the discovery and synthesis of quantum dots"',
 4        metadata={
 5            'source': 'path/to/prize.json',
 6            'seq_num': 1,
 7            'id': '1029',
 8            'firstname': 'Moungi',
 9            'surname': 'Bawendi',
10            'share': '3'
11        }
12    ),
13    ...
14]

Note: The metadata_func is only required if you want to add your own arbitrary metadata. If that’s not a concern, you can omit it and use the jq_schema.

 1from langchain_community.document_loaders import JSONLoader
 2from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
 3from langchain_community.vectorstores import Chroma
 4
 5embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
 6
 7loader = JSONLoader(
 8    file_path="./prize.json",
 9    jq_schema=".prizes[].laureates[]?.motivation",
10)
11documents = loader.load()

Langchain Expression with Chroma DB JSON (RAG)

After exploring how to use JSON files in a vector store, let’s integrate Chroma DB using JSON data in a chain.

For the purposes of this code, I used OpenAI model and embeddings.

Install the following dependencies:

1pip install langchain langchain-community langchain-openai chromadb jq

Then run the following code:

 1from langchain.prompts import ChatPromptTemplate
 2
 3from langchain_community.vectorstores import Chroma
 4from langchain_community.document_loaders import JSONLoader
 5from langchain_community.embeddings import OpenAIEmbeddings
 6
 7from langchain_core.output_parsers import StrOutputParser
 8from langchain_core.runnables import RunnableLambda, RunnablePassthrough
 9
10from langchain_openai import ChatOpenAI, OpenAIEmbeddings
11
12embedding_function = OpenAIEmbeddings()
13
14loader = JSONLoader(file_path="./prize.json", jq_schema=".prizes[]", text_content=False)
15documents = loader.load()
16
17db = Chroma.from_documents(documents, embedding_function)
18retriever = db.as_retriever()
19
20template = """Answer the question based only on the following context:
21{context}
22
23Question: {question}
24"""
25prompt = ChatPromptTemplate.from_template(template)
26
27model = ChatOpenAI()
28
29chain = (
30    {"context": retriever, "question": RunnablePassthrough()}
31    | prompt
32    | model
33    | StrOutputParser()
34)
35
36query = "What year did albert einstein win the nobel prize?"
37print(chain.invoke(query))

Steps:

Use the OpenAIEmbeddings to create an embedding function
Load the JSON file
Instantiate the Chroma DB instance from the documents & embedding model
Create a prompt template with context and question variables
Create a chain using the ChatOpenAI model with a retriever
Invoke the chain with the question What year did albert einstein win the nobel prize?

Output:

1Albert Einstein won the Nobel Prize in the year 1921.

This tutorial only includes the basic functionality for Chroma DB. Please visit my Chroma DB guide where I walk step-by-step on how to use it for a more in-depth tutorial.

DynamoDB PartiQL guide

Thomas Taylor — Sat, 27 Jan 2024 22:30:00 -0500

AWS introduced a PartiQL in 2019 as a “query language for all your data” that resembles SQL-like expressions. In November 2020, it was natively integrated with DynamoDB.

In this post, we’ll explore using PartiQL with DynamoDB.

What is PartiQL

PartiQL is an open source initiative that is maintained by Amazon under the Apache 2.0 license. Its goal is to be a SQL-compatible query language for accessing relational, semi-structured, and nested data.

Many AWS services support PartiQL:

Note about DynamoDB

While PartiQL introduces SQL-like expressions for DynamoDB, DynamoDB is a NoSQL database. Optimizing your access patterns, understanding key structure, using secondary indexes, and ensuring you’re not scanning over large amounts of data is still required.

PartiQL cannot escape DynamoDB’s rigidness - a simple SELECT statement can mistakenly scan over the entire table’s data. Because of this, ensure you’re using WHERE clauses appropriately.

Creating a table using AWS CLI

Let’s first create a table using the many-to-many relationship model from AWS’s documentation.

 1aws dynamodb create-table \
 2    --table-name ExampleTable \
 3    --attribute-definitions \
 4        AttributeName=pk,AttributeType=S \
 5        AttributeName=sk,AttributeType=S \
 6    --key-schema \
 7        AttributeName=pk,KeyType=HASH \
 8        AttributeName=sk,KeyType=RANGE \
 9    --provisioned-throughput \
10        ReadCapacityUnits=5,WriteCapacityUnits=5

This creates a table with a hash key of pk and a range key of sk.

Inserting data using PartiQL

With our ExampleTable in AWS, let’s insert a single row of mock data using the INSERT INTO operator:

1INSERT INTO ExampleTable VALUE {'pk':'INVOICE#92551','sk':'INVOICE#92551','date':'2018-02-07'}

Using the AWS CLI

1aws dynamodb execute-statement \
2    --statement "INSERT INTO ExampleTable VALUE {'pk':'INVOICE#92551','sk':'INVOICE#92551','date':'2018-02-07'}"

Transactions using PartiQL

For the next bit, we’re going to use the transactions API. DynamoDB restricts all transactions to the same operation: either read or write. In our case, we’re going to write all the following records:

1aws dynamodb execute-transaction \
2    --transact-statements \
3        "{\"Statement\":\"INSERT INTO ExampleTable VALUE {'pk':'INVOICE#92551','sk':'BILL#4224663','date':'2018-02-07'}\"}" \
4        "{\"Statement\":\"INSERT INTO ExampleTable VALUE {'pk':'INVOICE#92551','sk':'BILL#4224687','date':'2018-01-09'}\"}" \
5        "{\"Statement\":\"INSERT INTO ExampleTable VALUE {'pk':'INVOICE#92552','sk':'INVOICE#92552','date':'2018-03-04'}\"}" \
6        "{\"Statement\":\"INSERT INTO ExampleTable VALUE {'pk':'INVOICE#92552','sk':'BILL#4224687','date':'2018-01-09'}\"}" \
7        "{\"Statement\":\"INSERT INTO ExampleTable VALUE {'pk':'BILL#4224663','sk':'BILL#4224663','date':'2017-12-03'}\"}"

Inserting data using PartiQL with parameters

For the last insertion, we’ll use the --parameters option. Like SQL, we can use the ? question mark syntax to denote a parameter.

1aws dynamodb execute-statement \
2    --statement "INSERT INTO ExampleTable VALUE {'pk':?,'sk': ?,'date':?}" \
3    --parameters '[{"S":"BILL#4224687"},{"S":"BILL#4224687"},{"S":"2018-01-09"}]'

Updating data using PartiQL

PartiQL only supports updating a single item at a time. Let’s update the BILL#4224663 item with a new attribute named foo with a value of bar.

1aws dynamodb execute-statement \
2    --statement "UPDATE ExampleTable SET foo='bar' WHERE pk='BILL#4224663' and sk='BILL#4224663'"

The statement above creates a new attribute named foo with a vlaue of bar on the item with a pk and an sk of BILL#4224663. Like the DynamoDB put-item API call, we must specify the hash and range key values in the WHERE clause.

The following error will happen if we don’t specify both:

1An error occurred (ValidationException) when calling the ExecuteStatement operation: Where clause does not contain a mandatory equality on all key attributes

Removing an attribute or field using PartiQL

PartiQL also supports removing attributes using the following syntax:

1aws dynamodb execute-statement \
2    --statement "UPDATE ExampleTable REMOVE foo WHERE pk='BILL#4224663' and sk='BILL#4224663'"

This removes the foo field we created in the last section.

Query examples using PartiQL

As documented by AWS, the supported SELECT syntax for PartiQL is the following:

1SELECT expression  [, ...]
2FROM table[.index]
3[ WHERE condition ] [ [ORDER BY key [DESC|ASC] , ...]

YOU MUST SPECIFY PARTITION KEYS IF YOU DO NOT WANT TO PERFORM A FULL TABLE SCAN.

Using our table with its data from the prior examples, let’s select BILL#4224663’s record:

1SELECT *
2FROM ExampleTable
3WHERE pk = 'BILL#4224663' AND sk = 'BILL#4224663'

AWS CLI:

1aws dynamodb execute-statement \
2    --statement "SELECT * FROM ExampleTable WHERE pk = 'BILL#4224663' AND sk = 'BILL#4224663'"

Output:

 1{
 2    "Items": [
 3        {
 4            "sk": {
 5                "S": "BILL#4224663"
 6            },
 7            "date": {
 8                "S": "2017-12-03"
 9            },
10            "pk": {
11                "S": "BILL#4224663"
12            }
13        }
14    ]
15}

We can additionally select multiple records:

1aws dynamodb execute-statement \
2    --statement "SELECT pk, sk FROM ExampleTable WHERE pk IN ['INVOICE#92551', 'INVOICE#92552']"

PartiQL will execute two DynamoDB queries on our behalf since it’s searching for two pk values.

Querying with begins_with in PartiQL

Similarly, we can use the DynamoDB begins_with functionality in PartiQL.

In the example below, we query for all bills associated with INVOICE#92551

1SELECT *
2FROM ExampleTable
3where pk = 'INVOICE#92551' AND begins_with(sk, 'BILL#')

AWS CLI:

1aws dynamodb execute-statement \
2    --statement "SELECT * FROM ExampleTable where pk = 'INVOICE#92551' AND begins_with(sk, 'BILL#')"

Output:

 1{
 2    "Items": [
 3        {
 4            "sk": {
 5                "S": "BILL#4224663"
 6            },
 7            "date": {
 8                "S": "2018-02-07"
 9            },
10            "pk": {
11                "S": "INVOICE#92551"
12            }
13        },
14        {
15            "sk": {
16                "S": "BILL#4224687"
17            },
18            "date": {
19                "S": "2018-01-09"
20            },
21            "pk": {
22                "S": "INVOICE#92551"
23            }
24        }
25    ]
26}

Protecting against scans using PartiQL

PartiQL, while powerful, can be dangerous.

Let’s try to find items with a date attribute that is greater than 2018-01-09.

1SELECT *
2FROM ExampleTable
3WHERE "date" > '2018-01-09'

AWS CLI:

1aws dynamodb execute-statement \
2    --statement "SELECT * FROM ExampleTable WHERE \"date\" > '2018-01-09'"

Output:

 1{
 2    "Items": [
 3        {
 4            "sk": {
 5                "S": "BILL#4224663"
 6            },
 7            "date": {
 8                "S": "2018-02-07"
 9            },
10            "pk": {
11                "S": "INVOICE#92551"
12            }
13        },
14        {
15            "sk": {
16                "S": "INVOICE#92551"
17            },
18            "date": {
19                "S": "2018-02-07"
20            },
21            "pk": {
22                "S": "INVOICE#92551"
23            }
24        },
25        {
26            "sk": {
27                "S": "INVOICE#92552"
28            },
29            "date": {
30                "S": "2018-03-04"
31            },
32            "pk": {
33                "S": "INVOICE#92552"
34            }
35        }
36    ]
37}

A seemingly harmless query just completed an entire table scan because it used a non-key attribute in the WHERE clause.

To protect against these kinds of queries, AWS recommends using a DENY IAM policy:

 1{
 2   "Version":"2012-10-17",
 3   "Statement":[
 4      {
 5         "Effect":"Deny",
 6         "Action":[
 7            "dynamodb:PartiQLSelect"
 8         ],
 9         "Resource":[
10            "arn:aws:dynamodb:us-east-1:123456789012:table/ExampleTable"
11         ],
12         "Condition":{
13            "Bool":{
14               "dynamodb:FullTableScan":[
15                  "true"
16               ]
17            }
18         }
19      },
20      {
21         "Effect":"Allow",
22         "Action":[
23            "dynamodb:PartiQLSelect"
24         ],
25         "Resource":[
26            "arn:aws:dynamodb:us-east-1:123456789012:table/ExampleTable"
27         ]
28      }
29   ]
30}

Deleting data using PartiQL

Like updating, PartiQL can only delete 1 item from a table at a time.

We must specify all the keys associated with an item:

1aws dynamodb execute-statement \
2    --statement "DELETE FROM ExampleTable WHERE pk='INVOICE#92551' AND sk='BILL#4224663'"

Conclusion

In this post, we went over how to insert, update, query, and delete data from DynamoDB using PartiQL. PartiQL is a fun way to query DynamoDB, but, as we discussed, can easily turn into a DynamoDB killer with full table scans.

Using type annotations with kwargs in Python

Thomas Taylor — Fri, 26 Jan 2024 03:40:00 -0500

Historically in Python, adding explicit typing for kwargs was not directly supported; however, Python 3.12 added that ability.

In this post, we’ll go over methods for adding types to the kwargs argument.

Adding type annotations to kwargs

Python 3.12 released a new method for explicitly typing kwargs: using a TypedDict + Unpack.

 1from typing import NotRequired, TypedDict, Unpack
 2
 3
 4class Parameters(TypedDict):
 5    foo: int
 6    bar: str
 7    baz: NotRequired[str]
 8
 9
10def some_function(**kwargs: Unpack[Parameters]) -> None:
11    ...
12
13
14some_function(foo=1, bar="qux")

Here’s what happened:

Parameters inherits from TypedDict
NotRequired represents an optional argument
kwargs is typed using the Unpack[Parameters] type

mypy respects the typing.

Adding type annotations before Python 3.12

Before Python 3.12, there was not a clear way to explicitly declare typing for kwargs. The closest solutions are covered below.

Using TypedDict

Instead of using kwargs, ditch it for a TypedDict. For example:

 1from typing import TypedDict
 2
 3
 4class Parameters(TypedDict):
 5    foo: int
 6    bar: str
 7    baz: str | None
 8
 9
10def some_function(params: Parameters) -> None:
11    ...
12
13
14some_function(Parameters(foo=1, bar="test", baz=None))

It’s not great, but it provides structured typing and feedback using mypy.

Using Any

Simply using Any will suffice with mypy as well:

1from typing import Any
2
3
4def some_function(**kwargs: Any) -> None:
5    ...
6
7
8some_function(foo="bar")

But this is obviously not type safe since it disables type checking.

Using a single type

If your use-case allows it, you may define one type for the entire kwargs argument:

1def some_function(**kwargs: str) -> None:
2    ...
3
4
5some_function(foo="bar", bar="qux")

Unfortunately, this locks you into having the same type supplied per each argument.

Bruno: The Postman alternative you are looking for

Thomas Taylor — Sun, 21 Jan 2024 22:50:00 -0500

Postman used to be my favorite API testing tool, but the company’s decision to phase out the final local collection storage feature, Scratchpad, in favor of mandatory cloud services, has changed that.

What is the best Postman alternative?

Finding an alternative means it must satisfy the following requirements:

Local collection storage
Git-friendly collection definitions for my team members
CLI that can invoke collections
Beautiful desktop GUI
No forced account login

with a nice-to-have being open-source.

With that in mind, I came across a service that fulfills all the requirements: bruno.

What is Bruno

Bruno is an open-source Postman alternative that aims to provide a seamless offline experience for testing APIs.

It also plays nicely with Git by using a markdown language, Bru, for defining collections.

Installing Bruno

For MacOS users, installing bruno is easy with brew:

1brew install bruno

It’s also supported in many other popular package managers:

1# On Windows via Chocolatey
2choco install bruno
3
4# On Windows via Scoop
5scoop bucket add extras
6scoop install bruno
7
8# On Linux via Snap
9snap install bruno

Migrating from Postman

If you’re planning to migrate from Postman to Bruno, there are import options for collections and environments. For me, it was as simple as exporting the collections/environments in JSON format and importing them.

Scripting in Bruno

Like Postman, Bruno offers scripting capabilities for automating your workflow. I do not use this feature in either solution, but there is more information about it in the Bruno documentation.

Using Bruno CLI

In this section, I will use the Cat Fact API as an example collection.

Installing Bruno CLI

Firstly, let’s install the Bruno CLI using npm:

1npm install -g @usebruno/cli

Define a collection for catfacts

I opened Bruno and created a new collection named Cat Facts API with a single GET request named Get Random Fact.

When creating my collection, Bruno prompted me to select a folder to store my collection in.

That folder had this structure:

1catfacts
2├── Get Random Fact.bru
3├── bruno.json
4└── environments
5    └── production.bru

Get Random Fact.bru had file contents like this:

 1meta {
 2  name: Get Random Fact
 3  type: http
 4  seq: 1
 5}
 6
 7get {
 8  url: {{baseUrl}}/fact
 9  body: none
10  auth: none
11}

bruno.json had file contents like this:

1{
2  "version": "1",
3  "name": "Cat Facts API",
4  "type": "collection"
5}

and the production.bru file looked like this:

1vars {
2  baseUrl: https://catfact.ninja
3}

Invoke a collection from the Bruno CLI

I navigated to the collection folder and ran the following command:

1bru run --env production

Output:

1Running Folder Recursively 
2
3Get Random Fact (200 OK) - 177 ms
4
5Requests:    1 passed, 1 total
6Tests:       0 passed, 0 total
7Assertions:  0 passed, 0 total
8Ran all requests - 177 ms

This executes every .bru file in the folder.

Bruno additionally gives the ability to run single requests:

1bru run Get\ Random\ Fact.bru --env production

Output:

1Running Request 
2
3Get Random Fact (200 OK) - 175 ms
4
5Requests:    1 passed, 1 total
6Tests:       0 passed, 0 total
7Assertions:  0 passed, 0 total
8Ran all requests - 175 ms

Tagging in AWS CDK

Thomas Taylor — Wed, 17 Jan 2024 00:45:00 -0500

Tags are key/value metadata that enable users to categorize their AWS infrastructure. This is useful for identifying groups of resources, monitoring cloud spending, creating reports, etc.

In this post, we’ll explore different methods for tagging infrastructure using the AWS CDK. The code samples are in Python, but the techniques are applicable to all languages.

Tagging constructs

Constructs are the basic “building blocks” of the AWS CDK. They act as components and encapsulate necessary information for CloudFormation. Fortunately, the AWS CDK provides an easy-to-use API.

The standard method for tagging resources is:

1Tags.of(construct).add("key", "value")

Tag resources in AWS CDK

As shown above, tagging an individual resource is easy.

 1from aws_cdk import App, Stack, Tags
 2from aws_cdk import aws_iam as iam
 3
 4
 5class MyStack(Stack):
 6    def __init__(self, scope: App, construct_id: str, **kwargs) -> None:
 7        super().__init__(scope, construct_id, **kwargs)
 8
 9        my_role = iam.Role(self, "role", assumed_by=iam.AnyPrincipal())
10        Tags.of(my_role).add("owner", "team@example.com")

A tag with “owner” and “team@example.com” was added to the role construct.

Tag stacks in AWS CDK

Tagging a stack will apply tags to the stack construct and all resources within it.

 1from aws_cdk import App, Tags
 2
 3from stack import MyStack
 4
 5app = App()
 6my_stack = MyStack(app, "Stack")
 7
 8Tags.of(my_stack).add("owner", "team@example.com")
 9
10app.synth()

A tag with “owner” and “team@example.com” was added to the stack construct and all constructs within the stack.

Tag apps using AWS CDK

Like the previous example, tagging an app will auto tag the sub-constructs as well.

 1from aws_cdk import App, Tags
 2
 3from stack import MyStack
 4
 5app = App()
 6my_stack = MyStack(app, "Stack")
 7
 8Tags.of(app).add("owner", "team@example.com")
 9
10app.synth()

A tag with “owner” and “team@example.com” was added to the app construct and all constructs within the app.

Tag stacks using AWS CDK CLI

If you prefer specifying a tag at cdk deploy time, the --tags flag is available:

1cdk deploy Stack \
2    --tags owner=team@example.com \
3    --tags department=engineering

With the command above, two tags were applied to a stack named Stack.

Tag stacks using AWS CDK cdk.json file

An alternative to the command line flag is using the cdk.json file to specify deploy time tags.

Simply add the tags key to your cdk.json file with your tags defined in an array:

 1{
 2  "app": "python3 app.py",
 3  "tags": [
 4    {
 5      "Key": "owner",
 6      "Value": "team@example.com"
 7    },
 8    {
 9      "Key": "department",
10      "Value": "engineering"
11    }
12  ]
13}

Duplicate tags in AWS CDK

Occasionally, there may be duplicate tags applied to the same resource.

To control which tags overwrite others, a priority parameter can be supplied to the add method. By default, all tags applied have a 100 priority.

 1from aws_cdk import App, Tags
 2
 3from stack import MyStack
 4
 5app = App()
 6my_stack = MyStack(app, "Stack")
 7
 8Tags.of(my_stack).add("owner", "team2@example.com", priority=101)
 9Tags.of(my_stack).add("owner", "team@example.com")
10
11app.synth()

If both stacks have a priority of 100 – the bottom-most tag on the construct tree wins. In this case, the 101 priority superseded the bottom-most call.

A guide for deploying CloudFormation with CLI using Rain

Thomas Taylor — Sun, 14 Jan 2024 15:35:00 -0500

Last year, I wrote a post describing the difference between aws cloudformation deploy and aws cloudformation create-stack. I concluded that the easiest method for deploying CloudFormation templates was cloudformation deploy since it handled change sets on your behalf.

My opinion has changed; I discovered a new tool named Rain that is maintained by AWS.

What is Rain

For folks needing to deploy raw CloudFormation templates, rain is your hero. The tool brings you a lot of power:

Allows input parameters
Showcases real-time updates for stack deployments
Filters deployment logs sensibly
Provides the ability to generate CloudFormation templates via AI
Manipulates stack sets
Formats template files

And many more features! As a previous user of CloudFormation, this tool appears amazing.

Installing Rain

There are a few options for installing Rain, but I’ll use the golang install for this tutorial.

1go install github.com/aws-cloudformation/rain/cmd/rain@latest

For MacOS users, brew is easy:

1brew install rain

The remaining releases are featured in their GitHub repository.

Build CloudFormation stacks using Rain

The build command accepts a prompt parameter to generate CloudFormation templates using AI. Let’s use it for generating a CloudFormation template with a S3 bucket.

1rain build -p "A s3 bucket that is secure"

Output:

 1AWSTemplateFormatVersion: 2010-09-09
 2Description: Create an S3 bucket
 3
 4Resources:
 5  MyS3Bucket:
 6    Type: AWS::S3::Bucket
 7    Properties:
 8      BucketName: my-secure-bucket
 9      AccessControl: Private
10      VersioningConfiguration:
11        Status: Enabled

Although the AccessControl property is a considered legacy, it’s great that the template was generated on my behalf!

In addition, the build property will generate a skeleton for resources as well:

1rain build AWS::Route53::CidrCollection

1rain build CidrCollection

This outputs a template with all the parameters and dummy CHANGEME values for a Route53 CIDR collection:

 1AWSTemplateFormatVersion: "2010-09-09"
 2
 3Description: Template generated by rain
 4
 5Resources:
 6  MyCidrCollection:
 7    Type: AWS::Route53::CidrCollection
 8    Properties:
 9      Locations:
10        - CidrList:
11            - CHANGEME
12          LocationName: CHANGEME
13      Name: CHANGEME
14
15Outputs:
16  MyCidrCollectionArn:
17    Value: !GetAtt MyCidrCollection.Arn
18
19  MyCidrCollectionId:
20    Value: !GetAtt MyCidrCollection.Id

If you prefer json, there’s a flag for that:

1rain build CidrCollection --json

If you supply the command an ambigious term like “collection”, it provides you a warning:

1rain build collection

Output:

1Ambiguous resource type 'collection'; could be any of:
2  AWS::DevOpsGuru::ResourceCollection
3  AWS::Location::GeofenceCollection
4  AWS::OpenSearchServerless::Collection
5  AWS::Rekognition::Collection
6  AWS::Route53::CidrCollection

Bootstrap using Rain

Before deploying stacks into an environment, let’s run the bootstrap command:

1rain bootstrap

Output:

1Rain needs to create an S3 bucket called 'rain-artifacts-012345678901-us-east-1'. Continue? (Y/n)

This command creates an artifacts bucket that rain references when deploying stacks.

It has the following naming convention:

1rain-artifacts-{accountId}-{regionName}

Deploying stacks using Rain

In this section, we’ll explore deploying templates with rain.

Deploying an S3 bucket

Using the S3 template rain build generated, let’s deploy it!

This is my current directory structure:

1project
2└── s3.yml

This is the command I’ll use to deploy the s3.yml with a stack name of s3-bucket-stack:

1rain deploy s3.yml s3-bucket-stack

If rain bootstrap was not executed before deploying, it’ll prompt you to do so.

My deployment failed with the following message:

1CloudFormation will make the following changes:
2Stack s3-bucket-stack:
3  + AWS::S3::Bucket MyS3Bucket
4Do you wish to continue? (Y/n)
5Deploying template 's3.yml' as stack 's3-bucket-stack' in us-east-1.
6Stack s3-bucket-stack: ROLLBACK_COMPLETE
7Messages:
8  - MyS3Bucket: Resource handler returned message: "my-secure-bucket already exists (Service: S3, Status Code: 0, Request ID: null)" (RequestToken: c4a528c1-2fa0-e75e-efb1-5cfacc2aebd6, HandlerErrorCode: AlreadyExists)
9failed deploying stack 's3-bucket-stack'

Since S3 is a global service, the bucket name my-secure-bucket exists in another account or region. I modified the BucketName to be how-wtf-rain-bucket and issued the same command:

1Deleted existing, empty stack.
2CloudFormation will make the following changes:
3Stack s3-bucket-stack:
4  + AWS::S3::Bucket MyS3Bucket
5Do you wish to continue? (Y/n)
6Deploying template 's3.yml' as stack 's3-bucket-stack' in us-east-1.
7Stack s3-bucket-stack: CREATE_COMPLETE
8Successfully deployed s3-bucket-stack

A couple of awesome things to note:

It handled the failure of the stack gracefully and gave me the exact issue
I deployed again and it took care of everything
I did this all without touching the AWS console.

I am impressed by this tool already!

Deploying a Lambda function

I want to test the rain pkg command with lambda artifacts. Like the cloudformation package command, we specify a lambda directory or handler and it’ll handle zipping and uploading it!

I created a file named lambda.yml with the following contents:

 1AWSTemplateFormatVersion: 2010-09-09
 2Description: Create an lambda function
 3
 4Resources:
 5  LambdaExecutionRole:
 6    Type: AWS::IAM::Role
 7    Properties:
 8      AssumeRolePolicyDocument:
 9        Version: '2012-10-17'
10        Statement:
11          - Effect: Allow
12            Principal:
13              Service:
14                - lambda.amazonaws.com
15            Action:
16              - sts:AssumeRole
17      ManagedPolicyArns:
18        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
19  MyLambdaFunction:
20    Type: AWS::Lambda::Function
21    Properties:
22      Code: lambdafunction/
23      FunctionName: MyLovelyFunction
24      Handler: index.handler
25      Role: !GetAtt LambdaExecutionRole.Arn 
26      Runtime: python3.12
27
28Outputs:
29  LambdaFunctionArn:
30    Description: The ARN of the Lambda function
31    Value: !GetAtt MyLambdaFunction.Arn

This stack creates two resources:

A lambda execution role with basic permissions granted by the AWSLambdaBasicExecutionRole policy
A lambda function with the lambdafunction/ contents

And outputs the lambda function arn.

I ran the fmt command with the --verify flag:

1rain fmt lambda.yml --verify

The --verify command is best saved for CI environments since it outputs a 0 or 1 depending if the formatting is correct.

Output:

1lambda.yml: would reformat

I ran it with the --write command so that it fixed them:

1rain fmt lambda.yml --write

Let’s run the pkg command with an output file named out.yml:

1rain pkg lambda.yml --output out.yml

Here is a snapshot of my directory structure after doing the pkg command:

1project
2├── lambda.yml
3├── lambdafunction
4│   └── index.py
5└── out.yml

Now let’s deploy the out.yml template:

1rain deploy out.yml lambda-function-stack

Output:

 1CloudFormation will make the following changes:
 2Stack lambda-function-stack:
 3  + AWS::IAM::Role LambdaExecutionRole
 4  + AWS::Lambda::Function MyLambdaFunction
 5Do you wish to continue? (Y/n)
 6Deploying template 'out.yml' as stack 'lambda-function-stack' in us-east-1.
 7Stack lambda-function-stack: CREATE_COMPLETE
 8  Outputs:
 9    LambdaFunctionArn: arn:aws:lambda:us-east-1:012345678901:function:MyLovelyFunction # The ARN of the Lambda function
10Successfully deployed lambda-function-stack

Using the LambdaFunctionArn output, let’s invoke the function to ensure everything worked correctly:

1aws lambda invoke --function-name MyLovelyFunction response.txt

Here is the resulting response.txt:

1"Hello world!"

View logs of deployments

Another powerful feature of rain is that it can display sensible logs for stack deployments.

Using log command

Using the lambda function stack from the prior section, we can inspect the logs using the log command:

1rain log lambda-function-stack

Output:

1No interesting log messages to display. To see everything, use the --all flag

Since there were no “interesting” log messages to display, let’s see them all!

1rain log lambda-function-stack --all

Output:

1Jan 14 20:08:52 lambda-function-stack/lambda-function-stack (AWS::CloudFormation::Stack) CREATE_COMPLETE
2Jan 14 20:08:50 lambda-function-stack/MyLambdaFunction (AWS::Lambda::Function) CREATE_COMPLETE
3Jan 14 20:08:43 lambda-function-stack/MyLambdaFunction (AWS::Lambda::Function) CREATE_IN_PROGRESS "Resource creation Initiated"
4Jan 14 20:08:42 lambda-function-stack/MyLambdaFunction (AWS::Lambda::Function) CREATE_IN_PROGRESS
5Jan 14 20:08:40 lambda-function-stack/LambdaExecutionRole (AWS::IAM::Role) CREATE_COMPLETE
6Jan 14 20:08:24 lambda-function-stack/LambdaExecutionRole (AWS::IAM::Role) CREATE_IN_PROGRESS "Resource creation Initiated"
7Jan 14 20:08:22 lambda-function-stack/LambdaExecutionRole (AWS::IAM::Role) CREATE_IN_PROGRESS
8Jan 14 20:08:19 lambda-function-stack/lambda-function-stack (AWS::CloudFormation::Stack) CREATE_IN_PROGRESS "User Initiated"
9Jan 14 20:08:12 lambda-function-stack/lambda-function-stack (AWS::CloudFormation::Stack) REVIEW_IN_PROGRESS "User Initiated"

Generating Gantt chart for deployment times

Another cool feature is viewing the Gantt chart for the deployment times of different resources:

rain log lambda-function-stack --chart > chart.html

Here is the resulting html page:

Destroying stacks using Rain

Using the prior sections’ stacks, let’s use the rm command to showcase the stack deletion process:

1rain rm lambda-function-stack

Output:

1Stack lambda-function-stack: CREATE_COMPLETE
2Are you sure you want to delete this stack? (y/N) y
3Successfully deleted stack 'lambda-function-stack'

and the S3 bucket stack with the -y so that we don’t have to confirm again:

1rain rm s3-bucket-stack -y

Output:

1Successfully deleted stack 's3-bucket-stack'

Other features

There are other features not covered in this guide:

Stack sets
Forecasting errors using rain forecast (experimental)
Using rain’s instrinsic functions & modules (experimental)

As previously stated, if you need to work with raw CloudFormation - give this tool a try!

Implement version control in DynamoDB

Thomas Taylor — Sat, 13 Jan 2024 12:30:00 -0500

Amazon DynamoDB is a fully managed service provided by AWS that enables developers to quickly store data for their applications. In this article, I will showcase how to implement version control in DynamoDB for recording changes to data over time.

What is version control in DynamoDB

DynamoDB does not support native version control on a per-item basis. If you need to record changes to your data over time, it must be handled via the application. Luckily, there is a paradigm that supports storing multiple versions of the same data: duplication.

How to implement versioning in DynamoDB

We have a few options for storing versioned data in DynamoDB. For the purposes of this tutorial, we will use a single table design: i.e., using a primary key and a sort key to manage multiple data types.

Creating a table with versioning in DynamoDB

For the remaining sections of this tutorial, we’ll leverage the same single table.

Create a table

To begin, let’s define a table using the AWS CLI:

 1aws dynamodb create-table \
 2    --table-name table \
 3    --attribute-definitions \
 4        AttributeName=PK,AttributeType=S \
 5        AttributeName=SK,AttributeType=S \
 6    --key-schema \
 7        AttributeName=PK,KeyType=HASH \
 8        AttributeName=SK,KeyType=RANGE \
 9    --provisioned-throughput \
10        ReadCapacityUnits=5,WriteCapacityUnits=5

I specified a primary key named PK and a sort key of SK.

Time versioning

Time-based versioning may be useful for applications that need to store the status of data at certain timed intervals.

Inserting example data

To showcase the power of DynamoDB, let’s insert some values for a file object with an identifier of 1.

1aws dynamodb put-item \
2    --table-name table \
3    --item '{"PK":{"S":"file#1"},"SK":{"S":"2024-01-13T11:25:27-05:00"}}'

and another:

1aws dynamodb put-item \
2    --table-name table \
3    --item '{"PK":{"S":"file#1"},"SK":{"S":"2024-01-13T11:32:13-05:00"}}'

and one more:

1aws dynamodb put-item \
2    --table-name table \
3    --item '{"PK":{"S":"file#1"},"SK":{"S":"2024-01-13T11:37:58-05:00"}}'

Query for latest versions

Because the timestamps are sortable, we can leverage DynamoDB to perform the following requests:

Grab the last 100 versions
Grab all the versions by a specific interval (year, month, day, etc.)

Grab the latest 100 versions (or up to the DynamoDB limits):

1aws dynamodb query \
2    --table-name table \
3    --key-condition-expression "PK=:pk" \
4    --expression-attribute-values '{":pk":{"S":"file#1"}}' \
5    --no-scan-index-forward

The --no-scan-index-forward flag is important to sort the records in descending order rather than the default of ascending.

Output:

 1{
 2    "Items": [
 3        {
 4            "PK": {
 5                "S": "file#1"
 6            },
 7            "SK": {
 8                "S": "2024-01-13T11:37:58-05:00"
 9            }
10        },
11        {
12            "PK": {
13                "S": "file#1"
14            },
15            "SK": {
16                "S": "2024-01-13T11:32:13-05:00"
17            }
18        },
19        {
20            "PK": {
21                "S": "file#1"
22            },
23            "SK": {
24                "S": "2024-01-13T11:25:27-05:00"
25            }
26        }
27    ],
28    "Count": 3,
29    "ScannedCount": 3,
30    "ConsumedCapacity": null
31}

Grab all versions by a specific interval:

Using the begins_with or between operators, we can query for specific dates.

In the case below, I want to query everything that starts with 2024-01-13T11:3:

1aws dynamodb query \
2    --table-name table \
3    --key-condition-expression "PK=:pk and begins_with(SK, :sk)" \
4    --expression-attribute-values '{":pk":{"S":"file#1"},":sk":{"S":"2024-01-13T11:3"}}' \
5    --no-scan-index-forward

Output:

 1{
 2    "Items": [
 3        {
 4            "PK": {
 5                "S": "file#1"
 6            },
 7            "SK": {
 8                "S": "2024-01-13T11:37:58-05:00"
 9            }
10        },
11        {
12            "PK": {
13                "S": "file#1"
14            },
15            "SK": {
16                "S": "2024-01-13T11:32:13-05:00"
17            }
18        }
19    ],
20    "Count": 2,
21    "ScannedCount": 2,
22    "ConsumedCapacity": null
23}

Number versioning

For applications that want to maintain a “latest” version with the ability to rollback to a prior version, a number-based versioning paradigm will be optimal.

Inserting example data

To showcase the power of DynamoDB, let’s insert some values for a file object with an identifier of 2 and 2 different versions.

The first item will be the metadata for file#2. This contains the attributes for the file#2 when the application needs to fetch the latest version with the appropriate values.

1aws dynamodb put-item \
2    --table-name table \
3    --item '{"PK":{"S":"file#2"},"SK":{"S":"metadata"},"version":{"S":"2"},"foo":{"S":"baz"}}'

The second item will contain version 1’s information.

1aws dynamodb put-item \
2    --table-name table \
3    --item '{"PK":{"S":"file#2"},"SK":{"S":"version#1"},"version":{"S":"1"},"foo":{"S":"bar"}}'

The third item will contain version 2’s information.

1aws dynamodb put-item \
2    --table-name table \
3    --item '{"PK":{"S":"file#2"},"SK":{"S":"version#2"},"version":{"S":"2"},"foo":{"S":"baz"}}'

For this method, we duplicate the attributes and values of version#2 onto the main metadata object.

Query for latest versions

Let’s query for all versions:

1aws dynamodb query \
2    --table-name table \
3    --key-condition-expression "PK=:pk and begins_with(SK, :sk)" \
4    --expression-attribute-values '{":pk":{"S":"file#2"},":sk":{"S":"version#"}}' \
5    --no-scan-index-forward

Output:

 1{
 2    "Items": [
 3        {
 4            "version": {
 5                "S": "2"
 6            },
 7            "SK": {
 8                "S": "version#2"
 9            },
10            "PK": {
11                "S": "file#2"
12            },
13            "foo": {
14                "S": "baz"
15            }
16        },
17        {
18            "version": {
19                "S": "1"
20            },
21            "SK": {
22                "S": "version#1"
23            },
24            "PK": {
25                "S": "file#2"
26            },
27            "foo": {
28                "S": "bar"
29            }
30        }
31    ],
32    "Count": 2,
33    "ScannedCount": 2,
34    "ConsumedCapacity": null
35}

The user decides they want version#1 to be the selected version for file#2. To satisfy the request perform the following steps:

Modify the metadata item’s version attribute to 1
Duplicate the version’s attributes onto the metadata item

1aws dynamodb put-item \
2    --table-name table \
3    --item '{"PK":{"S":"file#2"},"SK":{"S":"metadata"},"version":{"S":"1"},"foo":{"S":"bar"}}'

Next time we fetch the latest version it’ll point to version 1:

1aws dynamodb get-item \
2    --table-name table \
3    --key '{"PK":{"S":"file#2"},"SK":{"S":"metadata"}}'

Output:

 1{
 2    "Item": {
 3        "version": {
 4            "S": "1"
 5        },
 6        "SK": {
 7            "S": "metadata"
 8        },
 9        "PK": {
10            "S": "file#2"
11        },
12        "foo": {
13            "S": "bar"
14        }
15    }
16}

Create IAM role with multiple principals in AWS CDK

Thomas Taylor — Wed, 10 Jan 2024 01:40:00 -0500

I need to create an IAM role that is assumable by multiple principals, but the AWS CDK role creation documentation only lists the following example:

 1lambda_role = iam.Role(self, "Role",
 2    assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"),
 3    description="Example role..."
 4)
 5
 6stream = kinesis.Stream(self, "MyEncryptedStream",
 7    encryption=kinesis.StreamEncryption.KMS
 8)
 9
10stream.grant_read(lambda_role)

That works great, but how do I list multiple principals?

Specifying multiple principals for IAM role

Luckily, there is a solution: the CompositePrincipal.

How to use the composite principal with AWS CDK

The CompositePrincipal class enables developers to pass in variadic arguments of type IPrincipal.

In the snippet of code below, I

Create two roles that are assumable by the {"AWS": "*"} principal.
Create a third role that contains a CompositePrincipal for both of the previous roles using iam.ArnPrincipal
Showcase the role arns in a CloudFormation output

 1from os import path
 2
 3import aws_cdk as cdk
 4import aws_cdk.aws_iam as iam
 5
 6
 7class MultiplePrincipalsStack(cdk.Stack):
 8    def __init__(self, scope: cdk.App, construct_id: str, **kwargs) -> None:
 9        super().__init__(scope, construct_id, **kwargs)
10        one = iam.Role(self, "Role1", assumed_by=iam.AnyPrincipal())
11        two = iam.Role(self, "Role2", assumed_by=iam.AnyPrincipal())
12
13        three = iam.Role(
14            self,
15            "RoleWithMultiplePrincipals",
16            assumed_by=iam.CompositePrincipal(
17                iam.ArnPrincipal(one.role_arn), iam.ArnPrincipal(two.role_arn)
18            ),
19        )
20
21        cdk.CfnOutput(self, "RoleArn1", value=one.role_arn)
22        cdk.CfnOutput(self, "RoleArn2", value=two.role_arn)
23        cdk.CfnOutput(self, "RoleArn3", value=three.role_arn)

After running cdk deploy, here are the stack outputs:

1Outputs:
2stack.RoleArn1 = arn:aws:iam::123456789012:role/stack-Role13A5C70C1-2RU2vpUAiHMy
3stack.RoleArn2 = arn:aws:iam::123456789012:role/stack-Role291939BC6-4geKeo5cAu3w
4stack.RoleArn3 = arn:aws:iam::123456789012:role/stack-RoleWithMultiplePrincipals7CD0E21E-Rd9Rxpc2Ioqe

Testing the role assumption

Using the outputs from before, we can verify that everything worked correctly using the AWS CLI.

I assumed Role1 by using the aws sts assume-role command:

1aws sts assume-role \
2    --role-arn arn:aws:iam::123456789012:role/stack-Role13A5C70C1-2RU2vpUAiHMy \
3    --role-session-name test

I exported the credentials from the output then assumed RoleWithMultiplePrincipals:

1aws sts assume-role \
2    --role-arn arn:aws:iam::123456789012:role/stack-RoleWithMultiplePrincipals7CD0E21E-Rd9Rxpc2Ioqe \
3    --role-session-name test2

Everything worked!

Serverless FastAPI with AWS Lambda, API Gateway, and AWS CDK

Thomas Taylor — Sun, 07 Jan 2024 16:40:00 -0500

By the end of this article, you’ll have a Python app deployed with FastAPI using AWS Lambda and API Gateway that is serving requests. In addition, you will have the infrastructure defined in code.

What is AWS Lambda and API Gateway

AWS Lambda and Amazon API Gateway are two services offered by AWS that enable developers to quickly spin up infrastructure without the hassle of provisioning or maintaining servers. In our case, we’ll host the FastAPI app in a Lambda function and the API Gateway will route requests to it.

This approach is immediately scalable upon deployment and includes a generous free tier by AWS.

How to deploy FastAPI with AWS Lambda + API Gateway + AWS CDK

Follow the instructions below to successfully deploy an AWS CDK app that includes a lambda function and API Gateway.

Create a requirements.txt

Create a requirements.txt file with the following information:

1aws-cdk-lib==2.118.0
2aws-cdk.aws-lambda-python-alpha==2.118.0a0
3constructs>=10.0.0,<11.0.0
4fastapi==0.108.0
5uvicorn==0.25.0
6mangum==0.17.0

To serve FastAPI locally, I chose uvicorn. This is not mandatory if you do not care about testing locally. To adapt the FastAPI router for API Gateway, we will use mangum.

Install the dependencies

1pip3 install -r requirements.txt

Create /app folder with a main.py file

You can use your preferred file structure. Whether you use this:

 1project/
 2├── app
 3│   ├── __init__.py
 4│   ├── main.py
 5│   ├── dependencies.py
 6│   └── routers
 7│   │   ├── __init__.py
 8│   │   ├── items.py
 9│   │   └── users.py
10│   └── internal
11│       ├── __init__.py
12│       └── admin.py

or this

1project/
2├── app
3│   ├── __init__.py
4│   └── main.py

ensure that there’s a main.py at the root of the app folder.

For the purposes of this tutorial, we’ll use a app/main.py file to keep it simple for a small todo app:

 1from fastapi import FastAPI
 2from mangum import Mangum
 3
 4app = FastAPI()
 5
 6
 7@app.get("/tasks")
 8async def get_tasks():
 9    # database lookup goes here
10    return [{"id": "ptvWZ3", "text": "hello!"}, {"id": "cqDUr3", "text": "another!"}]
11
12
13@app.get("/")
14async def root():
15    return {"message": "Hello World!"}
16
17
18handler = Mangum(app, lifespan="off")

Create a stack.py with an API Gateway and a lambda router

 1from os import path
 2
 3import aws_cdk as cdk
 4import aws_cdk.aws_lambda as lambda_
 5import aws_cdk.aws_apigateway as apigateway
 6import aws_cdk.aws_lambda_python_alpha as python
 7
 8
 9class ApiStack(cdk.Stack):
10    def __init__(self, scope: cdk.App, construct_id: str, **kwargs) -> None:
11        super().__init__(scope, construct_id, **kwargs)
12
13        lambda_function = python.PythonFunction(
14            self,
15            "Function",
16            entry=path.join(path.dirname(__file__), "app"),
17            runtime=lambda_.Runtime.PYTHON_3_11,
18            index="main.py",
19        )
20
21        api = apigateway.LambdaRestApi(self, "API", handler=lambda_function)

NOTE: We are using a construct that is in alpha. This enables the AWS CDK to use Docker to bundle our Python requirements together.

This seems small, but we are leveraging the power of the AWS CDK and their L3 constructs. This CDK app will:

Bundle python requirements and create a .zip file
Create a lambda function with the handler as main.handler
Create an API Gateway
Add a resource path of {proxy+} to the API
Add an ANY method to the resource path that points to the lambda function
Enable a “proxy integration” for the lambda function

What is an API Gateway proxy integration?

For the FastAPI logic to work appropriately, we need to configure the API Gateway to forward full API requests to the lambda function. With this setting, the lambda function’s responsibility increases as it must fulfill the requirements of the API Gateway service.

Fortunately, mangum takes care of this.

Create a requirements.txt file for the /app folder

As of time of writing (Jan. 7th, 2024), the python alpha function entry folder must have the requirements.txt file in it. Create app/requirements.txt file with the following:

1fastapi==0.108.0
2mangum==0.17.0

Create infra.py

Create a file that provisions the CDK app. I named mine infra.py.

1import aws_cdk as cdk
2
3from stack import ApiStack
4
5app = cdk.App()
6ApiStack(app, "ApiStack")
7
8app.synth()

Create cdk.json

Ensure that the entry file matches the one you created before.

1{
2  "app": "python3 infra.py"
3}

The directory structure should look like this:

1project/
2├── app
3│   ├── main.py
4│   └── requirements.txt
5├── cdk.json
6├── infra.py
7├── requirements.txt
8└── stack.py

Deploy the stack

Before deploying, ensure that Docker is running.

1cdk deploy

After the stack is deployed, we’re given the API endpoint in a stack output:

1Outputs:
2ApiStack.APIEndpoint1793E782 = https://{api_id}.execute-api.us-east-1.amazonaws.com/prod/

Testing the proxy integration

We’ll use curl to test our API:

1curl https://{api_id}.execute-api.us-east-1.amazonaws.com/prod/

Output:

1{"message": "Hello world!"}

The /tasks endpoint:

1curl https://{api_id}.execute-api.us-east-1.amazonaws.com/prod/tasks

Output:

1[{"id":"ptvWZ3","text":"hello!"},{"id":"cqDUr3","text":"another!"}]

Testing locally

The top-level requirements.txt enables us to test locally with uvicorn:

1uvicorn app.main:app --reload

Using curl,

1curl http://127.0.0.1:8000/

Output:

1{"message":"Hello World!"}

Cleaning up

If you wish to destroy your CDK app, run the following command:

1cdk destroy

Conclusion

After following the prior steps, you will have an understanding of how to deploy a functioning API Gateway and Lambda function powered by FastAPI.

Enjoy building on this example!

Creating a serverless API using AWS Lambda Powertools and CDK

Thomas Taylor — Fri, 05 Jan 2024 03:05:00 -0500

Amazon API Gateway + AWS Lambda is a powerful duo for creating easy-to-deploy scalable infrastructure that is immediately accessible by thousands of users. For this post, we will create an API Gateway with an lambda proxy integration using the AWS Lambda Powertools library and the AWS CDK.

In just a few lines of code, we’ll have our API deployed and ready to go! This tutorial assumes you know what the Amazon API Gateway and AWS Lambda services are.

What is AWS Lambda Powertools

AWS Lambda Powertools is an SDK released by Amazon that aims to increase developer productiviy by bundling similar functionality together:

Tracing
Logging
Event handling
Typing / Validation
Middleware

and more!

With this SDK, we’ll build a single lambda integration that will handle all the routes of our API Gateway.

If you’ve used Flask, Spring Boot, express, etc., this following code snippet should look familiar:

 1from aws_lambda_powertools import Logger, Tracer
 2from aws_lambda_powertools.event_handler import APIGatewayRestResolver
 3from aws_lambda_powertools.logging import correlation_paths
 4from aws_lambda_powertools.utilities.typing import LambdaContext
 5from aws_lambda_powertools.utilities.data_classes import event_source
 6from aws_lambda_powertools.utilities.data_classes.api_gateway_authorizer_event import (
 7    APIGatewayAuthorizerRequestEvent,
 8)
 9
10tracer = Tracer()
11logger = Logger()
12app = APIGatewayRestResolver()
13
14
15@app.get("/pets")
16@tracer.capture_method
17def get_pets():
18    return {"pets": []}
19
20
21@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
22@tracer.capture_lambda_handler
23@event_source(data_class=APIGatewayAuthorizerRequestEvent)
24def lambda_handler(
25    event: APIGatewayAuthorizerRequestEvent, context: LambdaContext
26) -> dict:
27    return app.resolve(event.raw_event, context)

The APIGatewayRestResolver behaves like a router. The lambda_handler processes the API Gateway event then resolves the route using the information that was passed from the lambda context. We’ll reference this example when creating our own.

How to deploy an API Gateway with Proxy Lambda with Powertools

Follow the instructions below to deploy a Lambda Function with Powertools bundled for an API Gateway proxy integration.

Create a requirements.txt

For this tutorial, version 2.118.0 of the AWS CDK was used.

1aws-cdk-lib==2.118.0
2constructs>=10.0.0,<11.0.0

Install the dependencies

1pip3 install -r requirements.txt

Create /handler folder with an index.py file

1import json
2
3
4def handler(event, context):
5    return {"statusCode": 200, "body": json.dumps({"message": "Hello world!"})}

Create stack.py with an API Gateway and a lambda router

 1from os import path
 2
 3import aws_cdk as cdk
 4import aws_cdk.aws_lambda as lambda_
 5import aws_cdk.aws_apigateway as apigateway
 6
 7
 8class ApiStack(cdk.Stack):
 9    def __init__(self, scope: cdk.App, construct_id: str, **kwargs) -> None:
10        super().__init__(scope, construct_id, **kwargs)
11
12        lambda_function = lambda_.Function(
13            self,
14            "Function",
15            runtime=lambda_.Runtime.PYTHON_3_11,
16            handler="index.handler",
17            code=lambda_.Code.from_asset(path.join(path.dirname(__file__), "handler")),
18        )
19
20        api = apigateway.LambdaRestApi(self, "API", handler=lambda_function)

The AWS CDK provides “L3” constructs; constructs that abstract away configuration we otherwise explicitly need define. In the case above, the L3 construct named LambdaRestApi handles a few things:

API Gateway creation
Lambda handler configuration
Resource path {proxy+} creation
ANY method creation

Create app.py

1import aws_cdk as cdk
2
3from stack import ApiStack
4
5app = cdk.App()
6ApiStack(app, "ApiStack")
7
8app.synth()

Create cdk.json

1{
2  "app": "python3 app.py"
3}

The directory structure should look like this:

1project/
2├── app.py
3├── cdk.json
4├── handler
5│   └── index.py
6├── requirements.txt
7└── stack.py

Deploy the stack

1cdk deploy

After the stack is deployed, we’re given the API endpoint in a stack output:

1Outputs:
2ApiStack.APIEndpoint1793E782 = https://{api_id}.execute-api.us-east-1.amazonaws.com/prod/

Testing the proxy integration

At this point, we can send any HTTP request and always get back the same response:

1curl https://{api_id}.execute-api.us-east-1.amazonaws.com/prod/

Output:

1{"message": "Hello world!"}

Add Lambda Powertools lambda layer using AWS CDK + SAR

Rather than write our own routing function based on the API Gateway event, let’s use AWS Lambda Powertools.

I have two preferred options for adding the Lambda Powertools layer:

Using the AWS lambda layer construct provided by AWS (requires Docker)
Explicitly using the AWS managed layer

For the purposes of this tutorial, I’ll reference the lambda layer using the serverless application respository.

 1from os import path
 2
 3import aws_cdk as cdk
 4import aws_cdk.aws_lambda as lambda_
 5import aws_cdk.aws_apigateway as apigateway
 6import aws_cdk.aws_sam as sam
 7
 8POWERTOOLS_BASE_NAME = "AWSLambdaPowertools"
 9POWERTOOLS_VERSION = "2.30.2"
10POWERTOOLS_ARN = "arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer"
11
12
13class ApiStack(cdk.Stack):
14    def __init__(self, scope: cdk.App, construct_id: str, **kwargs) -> None:
15        super().__init__(scope, construct_id, **kwargs)
16
17        powertools_app = sam.CfnApplication(
18            self,
19            f"{POWERTOOLS_BASE_NAME}Application",
20            location={
21                "applicationId": POWERTOOLS_ARN,
22                "semanticVersion": POWERTOOLS_VERSION,
23            },
24        )
25
26        powertools_layer_arn = powertools_app.get_att(
27            "Outputs.LayerVersionArn"
28        ).to_string()
29
30        powertools_layer_version = lambda_.LayerVersion.from_layer_version_arn(
31            self, f"{POWERTOOLS_BASE_NAME}", powertools_layer_arn
32        )
33
34        lambda_function = lambda_.Function(
35            self,
36            "Function",
37            runtime=lambda_.Runtime.PYTHON_3_11,
38            handler="index.handler",
39            code=lambda_.Code.from_asset(path.join(path.dirname(__file__), "handler")),
40            layers=[powertools_layer_version],
41        )
42
43        api = apigateway.LambdaRestApi(self, "API", handler=lambda_function)

Using AWS Lambda Powertools

Now that the lambda is successfully configured for AWS Lambda Powertools, let’s experiment using a small todo app:

 1from aws_lambda_powertools import Logger, Tracer
 2from aws_lambda_powertools.event_handler import APIGatewayRestResolver
 3from aws_lambda_powertools.logging import correlation_paths
 4from aws_lambda_powertools.utilities.typing import LambdaContext
 5
 6tracer = Tracer()
 7logger = Logger()
 8app = APIGatewayRestResolver()
 9
10
11@app.get("/tasks")
12@tracer.capture_method
13def get_tasks():
14    # database lookup goes here
15    return [{"id": "ptvWZ3", "text": "hello!"}, {"id": "cqDUr3", "text": "another!"}]
16
17
18@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
19@tracer.capture_lambda_handler
20def handler(event: dict, context: LambdaContext) -> dict:
21    return app.resolve(event, context)

Now we can test the different routes:

1curl https://{api_id}.execute-api.us-east-1.amazonaws.com/prod/

Output:

1{"statusCode":404,"message":"Not found"}

and the real /tasks endpoint:

1curl https://{api_id}.execute-api.us-east-1.amazonaws.com/prod/tasks

Output:

1[{"id":"ptvWZ3","text":"hello!"},{"id":"cqDUr3","text":"another!"}]

Conclusion

At this point, you will have a functioning AWS CDK application with an API Gateway Lambda Proxy integration setup! While the /tasks code is rudimentary, it provides the foundation for expansion for your own API!

Happy building!

Lambda Bash custom runtime with AWS CDK

Thomas Taylor — Tue, 02 Jan 2024 11:25:00 -0500

For this post, I will extend the AWS documentation for ‘Building a custom runtime’ by additionally using the AWS CDK in Python to deploy the Bash custom runtime for AWS Lambda.

How to deploy a Lambda custom runtime using AWS CDK

Follow the instructions below to deploy an AWS Lambda Function with a Bash custom runtime!

Create a /handler folder with a bootstrap file

AWS provides a bootstrap file that creates the custom runtime. Here is that script:

 1#!/bin/sh
 2
 3set -euo pipefail
 4
 5# Initialization - load function handler
 6source $LAMBDA_TASK_ROOT/"$(echo $_HANDLER | cut -d. -f1).sh"
 7
 8# Processing
 9while true
10do
11  HEADERS="$(mktemp)"
12  # Get an event. The HTTP request will block until one is received
13  EVENT_DATA=$(curl -sS -LD "$HEADERS" "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
14
15  # Extract request ID by scraping response headers received above
16  REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
17
18  # Run the handler function from the script
19  RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")
20
21  # Send the response
22  curl "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response"  -d "$RESPONSE"
23done

The tutorial explains what this script does in-depth, but here’s the summary:

Initializes by loading the handler file provided by the lambda
Grabs the event data from the invocation event
Grabs the request id from the invocation header
Runs the handler, then passes the response to the lambda runtime API

For more information about the runtime process, check out the AWS documentation here that describes the process and the API.

I created a folder named /handler and added the bootstrap file into it.

Create a requirements.txt

For this tutorial, version 2.117.0 of the AWS CDK was used.

1aws-cdk-lib==2.117.0
2constructs>=10.0.0,<11.0.0

Install the dependencies

1pip3 install -r requirements.txt

Add index.sh file to /handler directory

1handler() {
2  EVENT_DATA=$1
3  echo "$EVENT_DATA" 1>&2;
4  RESPONSE="Echoing request: '$EVENT_DATA'"
5  echo $RESPONSE
6}

Create stack.py with the lambda function

 1from os import path
 2
 3import aws_cdk as cdk
 4import aws_cdk.aws_lambda as lambda_
 5
 6
 7class BashFunctionStack(cdk.Stack):
 8    def __init__(self, scope: cdk.App, construct_id: str, **kwargs) -> None:
 9        super().__init__(scope, construct_id, **kwargs)
10
11        lambda_function = lambda_.Function(
12            self,
13            "Function",
14            runtime=lambda_.Runtime.PROVIDED_AL2023,
15            handler="index.handler",
16            code=lambda_.Code.from_asset(path.join(path.dirname(__file__), "handler")),
17        )
18
19        cdk.CfnOutput(self, "FunctionName", value=lambda_function.function_name)

Create app.py

1import aws_cdk as cdk
2
3from stack import BashFunctionStack
4
5app = cdk.App()
6BashFunctionStack(app, "BashFunctionStack")
7
8app.synth()

Create cdk.json

1{
2  "app": "python3 app.py"
3}

The directory structure should look like this:

1project/
2├── app.py
3├── cdk.json
4├── handler
5│   ├── bootstrap
6│   └── index.sh
7├── requirements.txt
8└── stack.py

Deploy the stack

1cdk deploy

Because of the CfnOutput, the function name is an output:

1Outputs:
2BashFunctionStack.FunctionName = BashFunctionStack-Function76856677-TiMb7dPSe1lE

Invoke the function

If everything worked correctly, we can invoke the function using the AWS CLI:

1aws lambda invoke response.txt \
2    --function-name FUNCTION_NAME \
3    --payload '{"text":"Hello"}' \
4    --cli-binary-format raw-in-base64-out

and receive the following output:

1{
2    "StatusCode": 200,
3    "ExecutedVersion": "$LATEST"
4}

with a response.txt of:

1Echoing request: '{"text":"Hello"}'

Goatcounter for blog analytics

Thomas Taylor — Sun, 31 Dec 2023 22:55:00 -0500

Going into 2024, I wanted to minimize the footprint of how.wtf.

This included:

Removing ads
Removing giscus, the commenting system
Removing Google Analytics
Reducing the website’s size to less than 20kb
Changing the theme to be super minimal with a focus on content

I have nothing against Google Analytics; I simply wanted something with less impact on my website’s loading speeds. Additionally, GDPR compliance requires companies using Google Analytics to gain explicit user consent for data collection. Personally, I just want a simple analytics - not a full-blown suite of tools.

What is Goatcounter

After careful consideration, I chose Goatcounter. It’s a minimal, opensource analytics tool that is easy to configure, privacy-aware, and lightweight. Like Google Analytics, I easily integrated it using a single script:

1<script data-goatcounter="https://yoursite.goatcounter.com/count" async src="//gc.zgo.at/count.js">script>

It provides a streamlined interface that gives exactly what I’m looking for:

Why I chose Goatcounter

Philosophically, I prefer minimalism. Goatcounter met my preferences and provided a streamlined integration and set up process. After a few clicks to setup my account, I immediately had analytics flowing. There’s always the option to self-host in the future, which is awesome!

To summarize here are my top reasons:

Open-source
Dead simple UI
Small (~3.5kb script)
Privacy-first
Seemingly GDPR compliant
Self-hosted option
API is easy to use if I ever need to
JavaScript-free options such as pixel tracking or logfile parsing

How to get top used Linux commands

Thomas Taylor — Wed, 27 Dec 2023 18:30:00 -0500

Retrieving the top 10 used commands in Linux is simple using the history command.

What is the history command

The history command, as its name implies, is used to view previously executed commands. It’s API is minimal and easy to use.

How to view top used commands in Linux

Using a combination of history and awk, we can pull the top 10 commands like this:

1history |
2    awk '{CMD[$1]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' |
3    grep -v "./" |
4    sort -n -r |
5    head -n 10

Output:

 11842 26.4617% git
 2520 7.47019% export
 3389 5.58828% cd
 4262 3.76383% sfdx
 5257 3.692% npx
 6254 3.6489% npm
 7237 3.40468% nvim
 8234 3.36159% aws
 9227 3.26103% rm
10195 2.80132% poetry

The awk command is the magic sauce. The sort command is used to reverse sort the list then the head command prints the last 10 lines.

If you prefer aligned columns with a counter to the left, we can use nl and the column command:

1history | 
2    awk '{CMD[$1]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' |
3    grep -v "./" |
4    column -c 3 -s " " -t |
5    sort -n -r |
6    nl |
7    head -n 10

Output:

 1    1	1842  26.4351%    git
 2    2	520   7.46269%    export
 3    3	389   5.58266%    cd
 4    4	262   3.76005%    sfdx
 5    5	257   3.68829%    npx
 6    6	254   3.64524%    npm
 7    7	237   3.40126%    nvim
 8    8	234   3.35821%    aws
 9    9	227   3.25775%    rm
10    10	195   2.79851%    poetry

In my case, git, export, and cd are my most used commands from the terminal.

How to wait for all threads to finish in Python

Thomas Taylor — Sun, 24 Dec 2023 23:00:00 -0500

For a side project, I needed to wait for multiple threads to complete before proceeding with the next step. In other words, I needed threads to behave like the Promise.all() functionality of JavaScript.

Wait for threads in Python

In the sections below, we’ll discuss different methods for waitin on threads to complete in Python.

Wait for all threads to complete using start / join

In simple terms, we can create the threads, append them to a list, then start and join them.

For example:

 1import time
 2from threading import Thread
 3
 4
 5def work(interval_seconds):
 6    print(f"sleeping for {interval_seconds} seconds")
 7    time.sleep(interval_seconds)
 8    print(f"slept for {interval_seconds} seconds")
 9
10
11threads = [
12    Thread(target=work, args=(1,)),
13    Thread(target=work, args=(2,)),
14    Thread(target=work, args=(1,)),
15]
16
17for t in threads:
18    t.start()
19
20for t in threads:
21    t.join()

Output:

1sleeping for 1 seconds
2sleeping for 2 seconds
3sleeping for 1 seconds
4slept for 1 seconds
5slept for 1 seconds
6slept for 2 seconds

This works well and may be all you need; however, there are alternatives.

Wait for all threads to complete using a ThreadPoolExecutor

In Python 3.2, concurrent.futures was released. The purpose was to provide a simple high-level API for asynchronously executing callables.

There were two classes released:

ThreadPoolExecutor for performing asynchronous executions using threads
ProcessPoolExecutor for performing asynchronous executions using processes

Both implement the same interface, the Executor class, so the examples provided below will work for either. For the purposes of this tutorial, I’ll use the ThreadPoolExecutor.

Here is the same example from above, but using the ThreadPoolExecutor:

 1import time
 2from concurrent.futures import ThreadPoolExecutor
 3
 4
 5def work(interval_seconds):
 6    print(f"sleeping for {interval_seconds} seconds")
 7    time.sleep(interval_seconds)
 8    print(f"slept for {interval_seconds} seconds")
 9
10
11with ThreadPoolExecutor(max_workers=2) as e:
12    e.submit(work, 1)
13    e.submit(work, 2)
14    e.submit(work, 1)

Output:

1sleeping for 1 seconds
2sleeping for 2 seconds
3slept for 1 seconds
4sleeping for 1 seconds
5slept for 1 seconds
6slept for 2 seconds

The executor provides a .map function for convenience as well:

 1import time
 2from concurrent.futures import ThreadPoolExecutor
 3
 4
 5def work(interval_seconds):
 6    print(f"sleeping for {interval_seconds} seconds")
 7    time.sleep(interval_seconds)
 8    print(f"slept for {interval_seconds} seconds")
 9
10
11with ThreadPoolExecutor(max_workers=2) as e:
12    e.map(work, [1, 2, 3])

Wait for all threads to complete and return their results

Firing an forgetting is useful, but returning results was my use-case. Using ThreadPoolExecutor or ProcessPoolExecutor makes this real easy.

 1import time
 2from concurrent.futures import ThreadPoolExecutor, as_completed
 3
 4
 5def work(interval_seconds, order):
 6    print(f"sleeping for {interval_seconds} seconds")
 7    time.sleep(interval_seconds)
 8    print(f"slept for {interval_seconds} seconds")
 9    return f"task {order}"
10
11
12with ThreadPoolExecutor(max_workers=2) as e:
13    futures = []
14    for x in [(1, 1), (2, 2), (1, 3)]:
15        futures.append(e.submit(work, x[0], x[1]))
16
17for future in as_completed(futures):
18    print(future.result())

In the example, I added a new parameter named order that represents when the item was queued.

The first job takes 1 second and returns 1
The second job takes 2 seconds and returns 2
The third job takes 1 seconds and returns 3

Output:

1sleeping for 1 seconds
2sleeping for 2 seconds
3slept for 1 seconds
4sleeping for 1 seconds
5slept for 2 seconds
6slept for 1 seconds
7task 2
8task 1
9task 3

Wait for all threads to complete and return their results in order

In the last example, the results where returned out of order. Just a small tweak enables the futures to be in order:

 1import time
 2from concurrent.futures import ThreadPoolExecutor, as_completed
 3
 4
 5def work(interval_seconds, order):
 6    print(f"sleeping for {interval_seconds} seconds")
 7    time.sleep(interval_seconds)
 8    print(f"slept for {interval_seconds} seconds")
 9    return f"task {order}"
10
11
12with ThreadPoolExecutor(max_workers=2) as e:
13    futures = []
14    for x in [(1, 1), (2, 2), (1, 3)]:
15        futures.append(e.submit(work, x[0], x[1]))
16
17for future in futures:
18    print(future.result())

Output:

1sleeping for 1 seconds
2sleeping for 2 seconds
3slept for 1 seconds
4sleeping for 1 seconds
5slept for 2 seconds
6slept for 1 seconds
7task 1
8task 2
9task 3

Promise.all and Promise.allSettled using threads

For the last topic, I want to replicate the interface that NodeJS provides: Promise.all() and Promise.allSettled().

If you’re unfamiliar, await Promise.all() waits for all the fulfilled promises to complete. If a rejection occurs, it throws an exception. Promise.allSettled() waits for all promises to be rejected or fulfilled without throwing an exception.

To replicate the functionality using concurrent.futures, I created a Job dataclass that has three properties:

The Callable (ie. function) that will be called for the job
The Tuple of positional arguments that will be passed to the job
The dict[str, Any] of keyword arguments that will be passed to the job

 1import concurrent.futures
 2import time
 3
 4from dataclasses import dataclass
 5from typing import Callable, Any, Tuple, Optional
 6
 7
 8def work(interval_seconds, order):
 9    print(f"sleeping for {interval_seconds} seconds")
10    time.sleep(interval_seconds)
11    print(f"slept for {interval_seconds} seconds")
12    return f"task {order}"
13
14
15@dataclass
16class Job:
17    func: Callable[..., Any]
18    args: Tuple[Any, ...] = ()
19    kwargs: dict[str, Any] = None
20
21    def execute(self):
22        return self.func(*self.args, **(self.kwargs or {}))
23
24
25def all(jobs):
26    with concurrent.futures.ThreadPoolExecutor() as executor:
27        futures = [executor.submit(job.execute) for job in jobs]
28        return [f.result() for f in futures]
29
30
31jobs = [
32    Job(func=work, args=(1, 1)),
33    Job(func=work, args=(2, 2)),
34    Job(func=work, args=(1, 3)),
35]
36
37results = all(jobs)
38print(results)

The all(jobs) function accepts a list of jobs to complete.

Output:

1sleeping for 1 seconds
2sleeping for 2 seconds
3sleeping for 1 seconds
4slept for 1 seconds
5slept for 1 seconds
6slept for 2 seconds
7['task 1', 'task 2', 'task 3']

We can extend the prior example by adding a all_settled function and a JobResult dataclass:

 1import concurrent.futures
 2import time
 3
 4from dataclasses import dataclass
 5from typing import Callable, Any, Tuple, Optional
 6
 7
 8def work(interval_seconds, order):
 9    print(f"sleeping for {interval_seconds} seconds")
10    time.sleep(interval_seconds)
11    print(f"slept for {interval_seconds} seconds")
12    return f"task {order}"
13
14
15@dataclass
16class JobResult:
17    status: str
18    value: Optional[Any] = None
19    reason: Optional[Exception] = None
20
21
22@dataclass
23class Job:
24    func: Callable[..., Any]
25    args: Tuple[Any, ...] = ()
26    kwargs: dict[str, Any] = None
27
28    def execute(self):
29        return self.func(*self.args, **(self.kwargs or {}))
30
31
32def all(jobs):
33    with concurrent.futures.ThreadPoolExecutor() as executor:
34        futures = [executor.submit(job.execute) for job in jobs]
35        return [f.result() for f in futures]
36
37
38def all_settled(jobs):
39    with concurrent.futures.ThreadPoolExecutor() as executor:
40        futures = [executor.submit(job.execute) for job in jobs]
41        concurrent.futures.wait(futures)
42
43        results = []
44        for future in futures:
45            if future.exception():
46                results.append(JobResult(status="rejected", reason=future.exception()))
47            else:
48                results.append(JobResult(status="fulfilled", value=future.result()))
49        return results
50
51
52jobs = [
53    Job(func=work, args=(1, 1)),
54    Job(func=work, args=(2, 2)),
55    Job(func=work, args=(1, 3)),
56]
57
58results = all_settled(jobs)
59for r in results:
60    print(r)

Output:

1sleeping for 1 seconds
2sleeping for 2 seconds
3sleeping for 1 seconds
4slept for 1 seconds
5slept for 1 seconds
6slept for 2 seconds
7JobResult(status='fulfilled', value='task 1', reason=None)
8JobResult(status='fulfilled', value='task 2', reason=None)
9JobResult(status='fulfilled', value='task 3', reason=None)

Stable Diffusion with Amazon Bedrock Python boto3

Thomas Taylor — Fri, 22 Dec 2023 02:00:00 -0500

Amazon Bedrock is a managed service provided by AWS that allows users to invoke models. For more information about the service, please refer to their user guide here.

For this article, we’re going to focus on how to invoke the Stable Diffusion model using the Python SDK: boto3. If you want to learn about the Amazon Bedrock SDK in general or how to invoke Claude with it, I have an article here that goes into detail.

What is Stable Diffusion

Stable diffusion is a text-to-image model first released in 2022. The term “diffusion” originates from diffusion models in machine learning. OpenAI’s DALL-E 2 is another example of a model that leverages diffusion. It’s open source, free, and easy to run! In addition, it has an active community and a plethora of how-to tutorials.

For more information about Stable Diffusion, check out the AWS write up about it.

How use Stable Diffusion with Amazon Bedrock

For this article, we’ll be leveraging the Python AWS SDK: boto3 to call the Stable Diffusion model.

Install boto3

Let’s begin by installing the latest version of boto3 and the Python image library PIL.

1pip3 install boto3 Pillow

Lookup the model inference parameters

Each model has specific inference parameters that must be supplied to Bedrock.

As of time of writing, Dec. 22nd 2023, the supported inference parameters for Stable Diffusion 1.0 text-to-image are:

 1{
 2    "text_prompts": [
 3        {
 4            "text": string,
 5            "weight": float
 6        }
 7    ],
 8    "height": int,
 9    "width": int,
10    "cfg_scale": float,
11    "clip_guidance_preset": string,
12    "sampler": string,
13    "samples",
14    "seed": int,
15    "steps": int,
16    "style_preset": string,
17    "extras": JSON object     
18}

The smallest payload we can send is this:

1{
2    "text_prompts": [{"text": string}]
3}

Use the invoke_model API

 1import base64
 2import json
 3import io
 4from PIL import Image
 5
 6import boto3
 7
 8client = boto3.client("bedrock-runtime", region_name="us-east-1")
 9
10body = {"text_prompts": [{"text": "A blue bird"}]}
11
12response = client.invoke_model(
13    body=json.dumps(body), modelId="stability.stable-diffusion-xl"
14)
15
16response_body = json.loads(response["body"].read())
17finish_reason = response_body.get("artifacts")[0].get("finishReason")
18
19if finish_reason in ["ERROR", "CONTENT_FILTERED"]:
20    raise Exception(f"Image error: {finish_reason}")
21
22base64_image = response_body["artifacts"][0]["base64"]
23base64_bytes = base64_image.encode("ascii")
24image_bytes = base64.b64decode(base64_bytes)
25image = Image.open(io.BytesIO(image_bytes))
26image.show()

Here is the line-by-line breakdown of the code above:

Instantiate the boto3.client("bedrock-runtime") client with a region_name of us-east-1.
json.dumps a dictionary of Stable Diffusion’s required inference parameters
Invoke the model by specifying the body (string) and modelId
If the finishReason is ERROR or CONTENT_FILTERED, raise an error
Convert the response_body to the necessary types for PIL.Image
Use image.show() to display a simple gui with the image

Output:

Invoke Amazon Bedrock models from Step Functions with AWS CDK

Thomas Taylor — Wed, 20 Dec 2023 02:30:00 -0500

AWS released an optimized integration for Amazon Bedrock. It allows StepFunction statemachines to directly call the Amazon Bedrock API without needing to write a Lambda function.

How to deploy a StepFunction with Amazon Bedrock integration using AWS CDK

Follow the instructions below to deploy a StepFunction with Amazon Bedrock integration using the AWS CDK. The following tutorial uses Python, but the same principles can be used with any language.

Create a requirements.txt

For this tutorial, version 2.115.0 of the AWS CDK was used.

1aws-cdk-lib==2.115.0
2constructs>=10.0.0,<11.0.0

Install the dependencies

1pip3 install -r requirements.txt

Create stack.py with the step function and bedrock task

For this tutorial, I opted to use the Claude V1 instant model. If you’re unsure where the sfn.TaskInput.from_object() parameters originate from, please review my article describing how to invoke Bedrock models directly.

 1from aws_cdk import (
 2    App,
 3    CfnOutput,
 4    Stack,
 5    aws_stepfunctions as sfn,
 6    aws_stepfunctions_tasks as tasks,
 7    aws_bedrock as bedrock,
 8)
 9
10
11class StepFunctionBedrockStack(Stack):
12    def __init__(self, scope: App, construct_id: str, **kwargs) -> None:
13        super().__init__(scope, construct_id, **kwargs)
14
15        model = bedrock.FoundationModel.from_foundation_model_id(
16            self, "Model", bedrock.FoundationModelIdentifier.ANTHROPIC_CLAUDE_INSTANT_V1
17        )
18        invoke_model_task = tasks.BedrockInvokeModel(
19            self,
20            "InvokeModel",
21            model=model,
22            body=sfn.TaskInput.from_object(
23                {
24                    "prompt": "\n\nHuman:Give me a cowboy joke\n\nAssistant:",
25                    "max_tokens_to_sample": 256,
26                }
27            ),
28            result_selector={"joke": sfn.JsonPath.string_at("$.Body.completion")},
29        )
30        state_machine = sfn.StateMachine(
31            self,
32            "StateMachine",
33            definition_body=sfn.DefinitionBody.from_chainable(invoke_model_task),
34        )
35
36        CfnOutput(self, "StateMachineArn", value=state_machine.state_machine_arn)

The code above creates a BedrockInvokeModel task with a body that’s specific for Anthropic Claude’s input and a result_selector that extracts the completion.

Create app.py

1import aws_cdk as cdk
2
3from stack import StepFunctionBedrockStack
4
5app = cdk.App()
6StepFunctionBedrockStack(app, "StepFunctionBedrockStack")
7
8app.synth()

Create cdk.json

1{
2  "app": "python3 app.py"
3}

The directory structure should look like this:

1project/
2├── app.py
3├── cdk.json
4├── requirements.txt
5└── stack.py

Deploy the stack

1cdk deploy

Because of the CfnOutput, the statemachine’s arn is in an output:

1Outputs:
2StepFunctionBedrockStack.StateMachineArn = arn:aws:states:us-east-1:0123456789101:stateMachine:StateMachine2E01A3A5-GeJycj800pUN

Start a statemachine execution

With the statemachine name, we can start an execution using the AWS CLI:

1aws stepfunctions start-execution  \
2    --state-machine-arn

Output:

1{
2    "executionArn": "arn:aws:states:us-east-1:0123456789101:execution:StateMachine2E01A3A5-GeJycj800pUN:387b0976-feb6-4c8c-9abe-af427546df3c",
3    "startDate": "2023-12-20T01:48:56.396000-05:00"
4}

then, we can describe the execution using the execution arn:

1aws stepfunctions describe-execution \
2    --execution-arn

Output:

 1{
 2    "executionArn": "arn:aws:states:us-east-1:0123456789101:execution:StateMachine2E01A3A5-GeJycj800pUN:387b0976-feb6-4c8c-9abe-af427546df3c",
 3    "stateMachineArn": "arn:aws:states:us-east-1:0123456789101:stateMachine:StateMachine2E01A3A5-GeJycj800pUN",
 4    "name": "387b0976-feb6-4c8c-9abe-af427546df3c",
 5    "status": "SUCCEEDED",
 6    "startDate": "2023-12-20T01:48:56.396000-05:00",
 7    "stopDate": "2023-12-20T01:48:57.814000-05:00",
 8    "input": "{}",
 9    "inputDetails": {
10        "included": true
11    },
12    "output": "{\"joke\":\" Here's one: Why don't seagulls fly over the bay? Because then they'd be bagels!\"}",
13    "outputDetails": {
14        "included": true
15    },
16    "redriveCount": 0,
17    "redriveStatus": "NOT_REDRIVABLE",
18    "redriveStatusReason": "Execution is SUCCEEDED and cannot be redriven"
19}

Pass information from the statemachine input to Bedrock

If you want to pass the input to the foundational model, you can do this using a combination of:

sfn.JsonPatch.string_at() and States.Format intrinsic function

 1from aws_cdk import (
 2    App,
 3    CfnOutput,
 4    Stack,
 5    aws_stepfunctions as sfn,
 6    aws_stepfunctions_tasks as tasks,
 7    aws_bedrock as bedrock,
 8)
 9
10
11class StepFunctionBedrockStack(Stack):
12    def __init__(self, scope: App, construct_id: str, **kwargs) -> None:
13        super().__init__(scope, construct_id, **kwargs)
14
15        model = bedrock.FoundationModel.from_foundation_model_id(
16            self, "Model", bedrock.FoundationModelIdentifier.ANTHROPIC_CLAUDE_INSTANT_V1
17        )
18        invoke_model_task = tasks.BedrockInvokeModel(
19            self,
20            "InvokeModel",
21            model=model,
22            body=sfn.TaskInput.from_object(
23                {
24                    "prompt": sfn.JsonPath.string_at(
25                        "States.Format('\n\nHuman:{}\n\nAssistant:', $$.Execution.Input.prompt)"
26                    ),
27                    "max_tokens_to_sample": 256,
28                }
29            ),
30            result_selector={"joke": sfn.JsonPath.string_at("$.Body.completion")},
31        )
32        state_machine = sfn.StateMachine(
33            self,
34            "StateMachine",
35            definition_body=sfn.DefinitionBody.from_chainable(invoke_model_task),
36        )
37
38        CfnOutput(self, "StateMachineArn", value=state_machine.state_machine_arn)

Now we can start the statemachine execution with a prompt:

1aws stepfunctions start-execution  \
2    --state-machine-arn  \
3    --input '{"prompt": "Write me a joke"}'

Amazon Bedrock Runtime examples using boto3

Thomas Taylor — Sun, 17 Dec 2023 00:55:00 -0500

Amazon Bedrock is a managed service provided by AWS that provides foundational models at your fingertips through a unified API. The service offers a range of features including foundational model invocations, fine-tuning, agents, guardrails, knowledge base searching, and more!

To read more about the service offerings, refer to its documentation.

What is Amazon Bedrock Runtime?

For this article, we’ll dive into using the Python AWS SDK, boto3, to call foundational models. Amazon Bedrock Runtime is the API entry point for invoking foundational models.

As of time of writing, Dec. 16th 2023, the supported API actions are:

InvokeModel
InvokeModelWithResponseStream

How to call models using Python boto3

Installing boto3

Invoking a foundational model using the InvokeModel API call is easy in Python! To begin, ensure that you have boto3 installed:

1pip3 install boto3

How to use the invoke_model method

Let’s instantiate the Amazon Bedrock Runtime client using boto3:

1import boto3
2
3client = boto3.client("bedrock-runtime")

then lookup the inference parameters needed for the body here. For the sake of this example, I’m invoking the Anthropic Claude model. Its inference parameters are:

1{
2    "prompt": "\n\nHuman:\n\nAssistant:",
3    "temperature": 0.5,
4    "top_p": 1,
5    "top_k": 250,
6    "max_tokens_to_sample": 200,
7    "stop_sequences": ["\n\nHuman:"]
8}

The bare minimum request for Claude is:

1{
2    "prompt": "\n\nHuman:\n\nAssistant:",
3    "max_tokens_to_sample": 200
4}

The body passed into the client must be a file or bytes. For Claude, the contentType as application/json.

 1import json
 2
 3import boto3
 4
 5client = boto3.client("bedrock-runtime", region_name="us-east-1")
 6
 7body = json.dumps(
 8    {
 9        "prompt": "\n\nHuman:What is your name?\n\nAssistant:",
10        "max_tokens_to_sample": 200,
11    }
12).encode()
13
14response = client.invoke_model(
15    body=body,
16    modelId="anthropic.claude-v2",
17    accept="application/json",
18    contentType="application/json",
19)
20
21response_body = json.loads(response["body"].read())
22print(response_body)

Here is the line-by-line breakdown of the code above:

Instantiate the boto3.client("bedrock-runtime") client with a region_name of us-east-1.
json.dumps a dictionary of Claude’s required inference parameters
Transform the JSON string into bytes using the .encode() method
Invoke the model by specifying the body (bytes), model id, accept, and contentType parameters.
Convert the StreamingBody to a JSON encoded string using .read() then turn the JSON string into a dictionary using json.loads

For more information about StreamingBody, refer to its documentation here. To keep it simple, we simply need to use the read method on the response body to get its contents.

Output:

1{'completion': ' My name is Claude.', 'stop_reason': 'stop_sequence', 'stop': '\n\nHuman:'}

The Claude JSON output includes a completion attribute with the text.

Here’s how to grab that information:

 1import json
 2
 3import boto3
 4
 5client = boto3.client("bedrock-runtime", region_name="us-east-1")
 6
 7body = json.dumps(
 8    {
 9        "prompt": "\n\nHuman:What is your name?\n\nAssistant:",
10        "max_tokens_to_sample": 200,
11    }
12).encode()
13
14response = client.invoke_model(
15    body=body,
16    modelId="anthropic.claude-v2",
17    accept="application/json",
18    contentType="application/json",
19)
20
21response_body = json.loads(response["body"].read())
22completion = response_body["completion"].strip()
23print(completion)

Output:

1My name is Claude.

How to stream models responses using Python boto3

Amazon Bedrock allows streaming LLM responses as well!

How to use the invoke_model with response stream method

Using the same example from above, let’s use the invoke_model_with_response_stream method:

 1import json
 2
 3import boto3
 4
 5client = boto3.client("bedrock-runtime", region_name="us-east-1")
 6
 7body = json.dumps(
 8    {
 9        "prompt": "\n\nHuman:Write me a 100 word essay about snickers candy bars\n\nAssistant:",
10        "max_tokens_to_sample": 200,
11    }
12).encode()
13
14response = client.invoke_model_with_response_stream(
15    body=body,
16    modelId="anthropic.claude-v2",
17    accept="application/json",
18    contentType="application/json",
19)
20
21stream = response["body"]
22if stream:
23    for event in stream:
24        chunk = event.get("chunk")
25        if chunk:
26            print(json.loads(chunk.get("bytes").decode()))

Output:

 1{'completion': ' Here', 'stop_reason': None, 'stop': None}
 2{'completion': ' is a 100 word essay', 'stop_reason': None, 'stop': None}
 3{'completion': ' about Snickers candy', 'stop_reason': None, 'stop': None}
 4{'completion': ' bars:\n\nS', 'stop_reason': None, 'stop': None}
 5{'completion': 'nickers is one of', 'stop_reason': None, 'stop': None}
 6{'completion': ' the most popular candy bars around. Introdu', 'stop_reason': None, 'stop': None}
 7{'completion': 'ced in 1930, it consists of nougat topped with', 'stop_reason': None, 'stop': None}
 8{'completion': ' caramel and peanuts that is encased in milk chocolate', 'stop_reason': None, 'stop': None}
 9{'completion': '. With its sweet and salty taste profile,', 'stop_reason': None, 'stop': None}
10{'completion': ' Snickers provides the perfect balance of flavors. The candy', 'stop_reason': None, 'stop': None}
11{'completion': " bar got its name from the Mars family's", 'stop_reason': None, 'stop': None}
12{'completion': ' favorite horse. Bite', 'stop_reason': None, 'stop': None}
13{'completion': ' into a Snickers and the rich', 'stop_reason': None, 'stop': None}
14{'completion': ' chocolate and caramel intermingle in your mouth while the', 'stop_reason': None, 'stop': None}
15{'completion': ' crunch of peanuts adds text', 'stop_reason': None, 'stop': None}
16{'completion': 'ural contrast. Loaded with sugar, Snick', 'stop_reason': None, 'stop': None}
17{'completion': 'ers gives you a quick burst of energy. It', 'stop_reason': None, 'stop': None}
18{'completion': "'s a classic candy bar that has endured for", 'stop_reason': None, 'stop': None}
19{'completion': ' decades thanks to its irresistible combination', 'stop_reason': None, 'stop': None}
20{'completion': ' of chocolate,', 'stop_reason': None, 'stop': None}
21{'completion': ' caramel, noug', 'stop_reason': None, 'stop': None}
22{'completion': "at and peanuts. Snickers' popularity shows", 'stop_reason': None, 'stop': None}
23{'completion': ' no signs of waning anytime soon.', 'stop_reason': 'stop_sequence', 'stop': '\n\nHuman:', 'amazon-bedrock-invocationMetrics': {'inputTokenCount': 21, 'outputTokenCount': 184, 'invocationLatency': 8756, 'firstByteLatency': 383}}

Instead of returning a StreamingBody like before, the response["body"] is an EventStream that can be iterated over in chunks.

ABC vs Protocol in Python

Thomas Taylor — Sat, 16 Dec 2023 12:00:00 -0500

Before typing was released for Python, the ABC class reigned as champion for describing shape and behaviors of classes. After type annotations, ABC and @abstractmethod were still used to describe the behaviors: they felt ‘interface-like’.

Then, Protocol was released and introduced a new way for declaring class behaviors.

Should you use ABC or Protocol?

Before describing why or why not you should choose one over the other, let’s delve into what each of them are.

What is ABC or Abstract Base Classes?

Abstract Base Classes, or ABC, is a a module that provides infrastructure to define abstract classes in Python.

As previously stated, they predated type hinting and were the go-to for enforcing class behaviors. Their goal was to provide a standardized means to test if an object adhered to a given specification.

For example,

 1from abc import ABC, abstractmethod
 2
 3
 4class Readable(ABC):
 5    @abstractmethod
 6    def read(self):
 7        pass
 8
 9
10class Stream(Readable):
11    def read(self):
12        return "read!"
13
14
15class FileReader(Readable):
16    def read(self):
17        return "some lines from a file"

ABCs use nominative subtyping by default: ie. you must explicitly inherit from a superclass to be considered a subtype.

Although, they do have a register method that enables a structural-like typing:

 1from abc import ABC, abstractmethod
 2
 3
 4class Readable(ABC):
 5    @abstractmethod
 6    def read(self):
 7        pass
 8
 9
10class Stream:
11    def read(self):
12        return "read!"
13
14
15Readable.register(Stream)

What is Protocol

Protocols are intended to make static checking easier - without needing to use an isinstance check at runtime.

They are treated as formalized duck typing - ie. the class only has to have the same attributes and methods… NOT be an instanceof.

Taking our class from earlier,

1from typing import Protocol
2
3
4class Readable(Protocol):
5    def read(self) -> str: ...

We can implement the Protocol in different ways:

1class Stream:
2    def read(self) -> str:
3        return "read!"

OR explicitly

1class Stream(Readable):
2    def read(self) -> str:
3        return "read!"

Additionally, we can enforce runtime evaluation using the runtime_checkable decorator:

 1from typing import Protocol, runtime_checkable
 2
 3
 4@runtime_checkable
 5class Readable(Protocol):
 6    def read(self) -> str: ...
 7
 8
 9class Stream(Readable):
10    def read(self) -> str:
11        return "read!"

Which is preferred?

From my perspective, Protocol wins. Given Python’s emphasis on duck typing (“if it walks and quacks like a duck, then it must be a duck”), it makes sense to opt for Protocol.

If there is a need to enforce runtime checks, it has that covered with the @runtime_checkable.

Protocol Pros:

Short syntax
Aligns with the “duck typing” mentality
Supports implicit or explicit declaration
Supports runtime enforcement

Protocol Cons:

Only available in Python 3.8 or above

How to use CSV files in vector stores with Langchain

Thomas Taylor — Tue, 12 Dec 2023 01:55:00 -0500

Retrieval-Augmented Generation (RAG) is a technique for improving an LLM’s response by including contextual information from external sources. In other terms, it helps a large language model answer a question by providing facts and information for the prompt.

For the purposes of this tutorial, we will implement RAG by leveraging a Chroma DB as a vector store with the FDIC Failed Bank List dataset.

Langchain with CSV data in a vector store

A vector store leverages a vector database, like Chroma DB, to fetch relevant documents using cosine similarity searches.

Install the dependencies:

1pip install langchain chromadb sentence-transformers

Use the following code:

 1from langchain.document_loaders import CSVLoader
 2from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
 3from langchain.vectorstores import Chroma
 4
 5embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
 6
 7loader = CSVLoader("./banklist.csv", encoding="windows-1252")
 8documents = loader.load()
 9
10db = Chroma.from_documents(documents, embedding_function)
11query = "Did a bank fail in North Carolina?"
12docs = db.similarity_search(query)
13print(docs[0].page_content)

Steps:

Use the SentenceTransformerEmbeddings to create an embedding function using the open source model of all-MiniLM-L6-v2 from huggingface.
Instantiate the loader for the csv files from the banklist.csv file. I had to use windows-1252 for the encoding of banklist.csv.
Load the files
Instantiate a Chroma DB instance from the documents & the embedding model
Perform a cosine similarity search
Print out the contents of the first retrieved document

Langchain Expression with Chroma DB CSV (RAG)

After exploring how to use CSV files in a vector store, let’s now explore a more advanced application: integrating Chroma DB using CSV data in a chain.

This section will demonstrate how to enhance the capabilities of our language model by incorporating RAG.

For the purposes of the following code, I opted for the OpenAI model and embeddings.

Install the dependencies:

1pip install langchain chromadb openai tiktoken

Use the following code:

 1from langchain.chat_models import ChatOpenAI
 2from langchain.document_loaders import CSVLoader
 3from langchain.embeddings import OpenAIEmbeddings
 4from langchain.prompts import ChatPromptTemplate
 5from langchain.vectorstores import Chroma
 6from langchain_core.output_parsers import StrOutputParser
 7from langchain_core.runnables import RunnableLambda, RunnablePassthrough
 8
 9embedding_function = OpenAIEmbeddings()
10
11loader = CSVLoader("./banklist.csv", encoding="windows-1252")
12documents = loader.load()
13
14db = Chroma.from_documents(documents, embedding_function)
15retriever = db.as_retriever()
16
17template = """Answer the question based only on the following context:
18{context}
19
20Question: {question}
21"""
22prompt = ChatPromptTemplate.from_template(template)
23
24model = ChatOpenAI()
25
26chain = (
27    {"context": retriever, "question": RunnablePassthrough()}
28    | prompt
29    | model
30    | StrOutputParser()
31)
32
33print(chain.invoke("What bank failed in North Carolina?"))

Borrowing from the prior example, we:

Created a prompt template with context and question variables
Created a chain using the ChatOpenAI model with a retriever
Invoked the chain with the question What bank failed in North Carolina?

Output:

1The bank that failed in North Carolina is Blue Ridge Savings Bank, Inc.

This tutorial only includes the basic functionality for Chroma DB. Please visit my Chroma DB guide where I walk step-by-step on how to use it for a more in-depth tutorial.

Open a GitHub pull request using the command line

Thomas Taylor — Sat, 09 Dec 2023 13:45:00 -0500

After pushing changes to a remote repository, a common next step is creating a pull request for your team members to review. In this article, we’ll discuss how to open a pull request from the command line using the GitHub CLI.

What is the GitHub CLI

The GitHub CLI is a tool that allows users to interact with GitHub directly from the terminal. It simplifies and streamlines GitHub workflows, enabling users to issue pull requests, check issues, clone repositories, etc., without leaving the terminal.

How to open a PR from the terminal

Opening a PR from the command line is straightforward using the gh CLI.

Install the GitHub CLI

To begin, install the GitHub CLI using the installation instructions.

For MacOS, the brew package manager is an easy option:

1brew install gh

Authenticate the GitHub CLI

After the GitHub CLI is installed, authenticate using the following command:

1gh auth login

If you’re having trouble, follow the instructions provided by the GitHub team here for more options.

Open a pull request from the terminal

The following opens an interactive experience for your current branch:

1gh pr create

Create a pull request in the browser from the terminal

If you prefer opening pull requests in the web version, the CLI can direct you to it:

1gh pr create --web

Create a pull request using a template from the terminal

If your organization requires using a PR template, this is supported as well:

1gh pr create --template

Create a draft pull request from the terminal

If you want to create a draft pull request, the GitHub CLI has you covered:

1gh pr create --draft

Create a pull request using a title and body from the terminal

The CLI supports explicitly defining the --title and --body:

1gh pr create --title "Some title" --body "Some body"

Create a pull request for a different branch from the terminal

You may want to create a PR for a different branch using the CLI. This is covered as well:

1gh pr create --base main --head feature-branch

Conclusion

To wrap up, using the GitHub CLI can significantly boost your workflow. I encourage you to explore this tool – it’s like having a superpower right in your terminal.

Set up time-to-live (TTL) attribute in DynamoDB

Thomas Taylor — Fri, 08 Dec 2023 01:50:00 -0500

Amazon DynamoDB’s Time to Live (TTL) feature allows for automatic item deletion after a specified timestamp. It comes at no extra cost and is useful for removing outdated data.

TTL use-cases

Remove stale data and save on DynamoDB memory usage
Retain sensitive data only up to contractual or regulatory obligations
Trigger processes using DynamoDB Streams based on TTL deletions since the deletion stream is marked as “system” rather than normal

How to setup TTL in DynamoDB step-by-step

This tutorial will use Python and the AWS CLI for demonstration purposes. The same logic can be applied to the tooling of your choice.

Create a DynamoDB table with TTL

To keep things simple, we’ll leverage the AWS CLI to quickly create a table.

 1aws dynamodb create-table \
 2    --table-name your-table \
 3    --attribute-definitions \
 4        AttributeName=pk,AttributeType=S \
 5        AttributeName=sk,AttributeType=S \
 6    --key-schema \
 7        AttributeName=pk,KeyType=HASH \
 8        AttributeName=sk,KeyType=RANGE \
 9    --provisioned-throughput \
10        ReadCapacityUnits=5,WriteCapacityUnits=5

then enable TTL

1aws dynamodb update-time-to-live \
2    --table-name your-table \
3    --time-to-live-specification \
4        AttributeName=expires_at,Enabled=true

Output:

1{
2    "TimeToLiveSpecification": {
3        "Enabled": true,
4        "AttributeName": "expires_at"
5    }
6}

If you desire to deploy a table via AWS CDK or CloudFormation, they allow you to specify a TimeToLiveSpecification upon table creation:

 1AWSTemplateFormatVersion: '2010-09-09'
 2Description: CloudFormation template for creating a DynamoDB table with TTL.
 3Resources:
 4  MyDynamoDBTable:
 5    Type: 'AWS::DynamoDB::Table'
 6    Properties:
 7      TableName: your-table
 8      AttributeDefinitions:
 9        - AttributeName: pk
10          AttributeType: S
11        - AttributeName: sk
12          AttributeType: S
13      KeySchema:
14        - AttributeName: pk
15          KeyType: HASH
16        - AttributeName: sk
17          KeyType: RANGE
18      ProvisionedThroughput:
19        ReadCapacityUnits: 5
20        WriteCapacityUnits: 5
21      TimeToLiveSpecification:
22        AttributeName: expires_at 
23        Enabled: true

Add TTL attribute with proper format to an item

The value for a TTL attribute is required to be in Unix Epoch time format. Items with a timestamp older than the current time, except those older by 5 years or more, are marked as expired and will be deleted by the per-partition background scanner.

Here is a Python example demonstrating how to insert an item with correctly formatted TTL:

 1import boto3
 2import time
 3
 4dynamodb = boto3.resource("dynamodb")
 5table = dynamodb.Table("your-table")
 6current_time = int(time.time())
 7
 8# TTL value (e.g., 24 hours from now)
 9ttl_value = current_time + 24 * 60 * 60  # 24 hours from now in seconds
10item = {"pk": "1", "sk": "entity", "expires_at": ttl_value}
11
12response = table.put_item(Item=item)
13print(response)

Output:

 1{
 2  "ResponseMetadata": {
 3    "RequestId": "...",
 4    "HTTPStatusCode": 200,
 5    "HTTPHeaders": {
 6      "server": "Server",
 7      "date": "Fri, 08 Dec 2023 06:24:48 GMT",
 8      "content-type": "application/x-amz-json-1.0",
 9      "content-length": "2",
10      "connection": "keep-alive",
11      "x-amzn-requestid": "...",
12      "x-amz-crc32": "..."
13    },
14    "RetryAttempts": 0
15  }
16}

DynamoDB Query without TTL expired items

Directly from the AWS documentation:

TTL typically deletes expired items within a few days. Depending on the size and activity level of a table, the actual delete operation of an expired item can vary. Because TTL is meant to be a background process, the nature of the capacity used to expire and delete items via TTL is variable (but free of charge).

DynamoDB queries and scans will return the items that are supposed to be expired if they have not been removed from the table. To remediate this, use a filter expression:

 1import boto3
 2import time
 3
 4dynamodb = boto3.resource("dynamodb")
 5table = dynamodb.Table("your-table")
 6current_time = int(time.time())
 7print(current_time)
 8
 9query_params = {
10    "KeyConditionExpression": "pk = :pk",
11    "FilterExpression": "attribute_not_exists(#ttl) OR #ttl > :now",
12    "ExpressionAttributeValues": {":pk": "1", ":now": current_time},
13    "ExpressionAttributeNames": {"#ttl": "expires_at"},
14}
15
16response = table.query(**query_params)
17print(response)

How to use NordVPN on Linux step-by-step guide

Thomas Taylor — Wed, 06 Dec 2023 02:20:00 -0500

In this article, we’ll discuss how to use NordVPN on Linux. This includes installation, authentication, and general usage of the CLI in Linux.

Please note, the links provided for NordVPN are affiliate links. This means if you click on them and make a purchase, I may receive a commission at no extra cost to you. This helps support the blog and allows me to continue creating content like this.

What is NordVPN?

NordVPN is a robust virtual private network (VPN) service that ensures secure and anonymized internet access. It’s easy to use, fast, and provides many servers to connect from. Truth be told, it has been my go-to VPN solution for years.

Why use NordVPN?

14 million users
5,400+ servers in 59 countries
Split tunneling support
24/7 customer support

Installing NordVPN on Linux

1sh <(curl -sSf https://downloads.nordcdn.com/apps/linux/install.sh)

If curl is not available on your system, wget is an option:

1sh <(wget -qO - https://downloads.nordcdn.com/apps/linux/install.sh)

If you receive the following error:

1Whoops! Permission denied accessing /run/nordvpn/nordvpnd.sock

then modify your current user’s permissions by running the following command:

1sudo usermod -aG nordvpn $USER

For permission refresh, either reboot your machine using

1sudo reboot now

or re-login using your current user:

1su - $USER

You’re all set!

To begin, you must login to your NordVPN account using the following command:

1nordvpn login

This outputs a login url. COPY the url and paste it into a browser of your choosing.

After a successful attempt, the Continue button should authenticate your terminal session.

To verify, run the following command:

1nordvpn account

If you receive the “You are not logged in” message, then you may optionally provide the callback URI to the CLI.

To do this, issue the nordvpn login command again. Before clicking the Continue button this time, right click it and copy the link.

Paste the uri in quotes as the value for the --callback argument:

1nordvpn login --callback "nordvpn://login?action=login?exchange_token=&status=done"

Connect with NordVPN

Like I mentioned earlier, this is my preferred VPN because of its ease-of-use. To connect, simply run the following command:

1nordvpn connect

Output:

1Connecting to the United States #10028 (us10028.nordvpn.com)
2You are connected to United States #10028 (us10028.nordvpn.com)

You will automatically be connected to the nearest location.

Connect with specific country using NordVPN

The NordVPN client allows connections to other locations seamlessly. To begin, run the following command to discover the countries available:

1nordvpn countries

Connect to the country:

1nordvpn connect COUNTRY

Connect with specific city using NordVPN

Similary, discover the cities available by country using the following command:

1nordvpn cities COUNTRY

For example,

1nordvpn cities United_States

Connect to a city:

1nordvpn connect COUNTRY CITY

For example,

1nordvpn connect United_States New_York

Automatically start NordVPN connection on Linux boot

To automatically connect NordVPN on boot, simply run the following command:

1nordvpn set autoconnect enabled

Set up NordVPN Kill Switch on Linux

If NordVPN connection is lost, you may optionally enable the killswitch to ensure all internet traffic stops:

1nordvpn set killswitch enabled

View NordVPN settings

If you want to view all the settings that can be toggled simply run:

1nordvpn settings

Conclusion

This article provides a step-by-step guide on installing and using NordVPN on Linux. It covers installation, logging in, connecting to different servers, and setting up features like auto-connect and killswitch. I hope you enjoyed the content!

How to use Docker without Docker Desktop on MacOS

Thomas Taylor — Fri, 01 Dec 2023 01:25:00 -0500

Docker Desktop was my de facto choice on MacOS due to its ease of installation and container management. This changed after Docker updated their subscription service agreement.

Step-by-step guide to installing Docker without Docker Desktop

The following tutorial assumes that you use brew as your package manager.

Install docker

Firstly, install docker and docker-credential-helper.

1brew install docker-credential-helper docker

docker-credential-helper provides a way for Docker to use the MacOS Keychain as a credential store.

Install colima

The true power comes from colima: a container runtime for MacOS and Linux.

Install it using brew:

1brew install colima

Start colima

Colima boasts its CLI ease of use! To get started, simply start the service:

1colima start

Using colima

After colima is installed, docker should hopefully work out-of-the-box:

1docker ps
2CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Some applications do not respect docker contexts and will yield the following error:

1Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

To remediate the issue, set the DOCKER_HOST variable.

1export DOCKER_HOST=unix://${HOME}/.colima/default/docker.sock

Golang AWS Lambda project structure

Thomas Taylor — Wed, 29 Nov 2023 01:45:00 -0500

AWS Lambda is an easy-to-use serverless offering that enables developers to quickly deploy code without worrying about maintenance, orchestration, scaling, etc. It’s simple to get started and its free tier is generous!

Golang project structure

GO does not have a recommended project structure. In fact, keeping a project structure flat is preferrable for small apps:

1project/
2├── go.mod
3├── go.sum
4├── user.go
5└── main.go

My personal preference is to use the flat structure for small apps or a combination of internal + cmd:

1project/
2├── cmd
3│   └── app1
4│       └── main.go
5├── go.mod
6├── go.sum
7└── internal
8    └── user.go

What is the internal module

The internal module can only be imported from packages rooted at the parent of the internal folder. In other words, the internal folder is “private” or non-existent outside the package. It reduces API surface area and establishes a clear domain boundary that is enforced by GO.

What is the cmd module

The cmd directory is a common convention I prefer for organizing sub-apps into runnable binaries. For example, a larger project may have multiple “apps” required to run it:

 1store/
 2├── cmd
 3│   ├── api
 4│   │   └── main.go
 5│   ├── frontend
 6│   │   └── main.go
 7│   └── worker
 8│       └── main.go
 9├── go.mod
10├── go.sum
11└── internal
12    ├── cart.go
13    ├── product.go
14    └── user.go

Each “app” within the cmd directory will have a distributable binary that can be invoked with required / optional flags. For example, the frontend binary may have a -port flag. The intention of the main.go files are to be THIN entry points that delegate the heavy processing to the internal layer.

It’s common for a simple main.go to look like this:

 1package main
 2
 3import (
 4    "github.com/user/repo/internal"
 5)
 6
 7func main() {
 8    c := internal.NewClient()
 9    c.DoSomething()
10}

Golang AWS Lambda project structure

Following the prior convention, lambdas can be similary nested in the cmd directory:

 1project/
 2├── cmd
 3│   ├── function1
 4│   │   └── main.go
 5│   └── function2
 6│       └── main.go
 7├── go.mod
 8├── go.sum
 9└── internal
10    └── user.go

With very THIN main.go entry handlers:

 1package main
 2
 3import (
 4    "github.com/aws/aws-lambda-go/lambda"
 5
 6    "github.com/user/repo/internal"
 7)
 8
 9func HandleRequest() error {
10    c := internal.NewClient()
11    c.DoSomething()
12}
13
14func main() {
15    lambda.Start(HandleRequest)
16}

If cmd doesn’t fit your use case, creating another folder is completely fine:

 1project/
 2├── lambda 
 3│   ├── function1
 4│   │   └── main.go
 5│   └── function2
 6│       └── main.go
 7├── go.mod
 8├── go.sum
 9└── internal
10    └── user.go

Even having separate cmd and lambda folders to distinguish between CLI-invocable apps and Lambda-invocable apps is fine.

The main point is keeping the main.go entry points thin.

Golang AWS Lambda project structure with CDK

For my use-case, I kept the lambda handlers nested within the cmd directoy. In addition, I created a deploy app for the CDK that contains the stack-creation.

 1project/
 2├── cdk.json
 3├── cmd
 4│   ├── function1
 5│   │   └── main.go
 6│   └── infrastructure
 7│       └── main.go
 8├── go.mod
 9├── go.sum
10└── internal
11    └── s3.go

The cdk.json is simple:

1{
2  "app": "go mod download && go run cmd/deploy/main.go"
3}

How to count Amazon Bedrock Anthropic tokens with Langchain

Thomas Taylor — Fri, 24 Nov 2023 01:00:00 -0500

Tracking Amazon Bedrock Claude token usage is simple using Langchain!

How to track tokens using Langchain

For OpenAI models, Langchain provides a native Callback handler for tracking token usage as documented here.

1from langchain.callbacks import get_openai_callback
2from langchain.llms import OpenAI
3
4llm = OpenAI(temperature=0)
5with get_openai_callback() as cb:
6    llm("What is the square root of 4?")
7
8total_tokens = cb.total_tokens
9assert total_tokens > 0

However, other model families do not have this luxury.

Counting Anthropic Bedrock Tokens

Anthropic released a Bedrock client for their models that provides a method for counting tokens. I wrote an entire article about how to use it here.

Under the hood, Langchain uses the Anthropic client within the Bedrock LLM class to provide token counting.

How to count tokens using Langchain Bedrock LLM Step-by-Step

Before starting, please review my post about using Bedrock with Langchain here. It describes how to use boto3 with langchain to easily instantiate clients. For the sake of this tutorial, I’ll assume you already have your AWS session configured.

Installation

Firstly, install the required libraries:

1pip3 install anthropic boto3 langchain

Counting Bedrock LLM tokens

To keep it simple, we’ll begin by importing the Bedrock LLM and counting the input and output tokens.

 1from langchain.llms import Bedrock
 2
 3import boto3
 4
 5client = boto3.client("bedrock-runtime")
 6
 7llm = Bedrock(client=client, model_id="anthropic.claude-instant-v1")
 8prompt = "who are you?"
 9print(llm.get_num_tokens(prompt))
10
11result = llm(prompt)
12print(result, llm.get_num_tokens(result))

Output:

14
2My name is Claude. I'm an AI assistant created by Anthropic. 16

In this case, 4 input tokens and 16 output tokens were used.

Counting Bedrock LLM tokens in chain

For my use case, I need to count the tokens used in chains. We’ll need to build a custom handler since Langchain does not provide native support for counting Bedrock tokens.

For more information on Callback handlers, please reference the Langchain documentation here.

In the code below, I extended the BaseCallbackHandler class with AnthropicTokenCounter. In the constructor, a Bedrock LLM client is required since it contains the token counting function.

 1from langchain.callbacks.base import BaseCallbackHandler
 2
 3import boto3
 4
 5
 6class AnthropicTokenCounter(BaseCallbackHandler):
 7    def __init__(self, llm):
 8        self.llm = llm
 9        self.input_tokens = 0
10        self.output_tokens = 0
11
12    def on_llm_start(self, serialized, prompts, **kwargs):
13        for p in prompts:
14            self.input_tokens += self.llm.get_num_tokens(p)
15
16    def on_llm_end(self, response, **kwargs):
17        results = response.flatten()
18        for r in results:
19            self.output_tokens = self.llm.get_num_tokens(r.generations[0][0].text)

The on_llm_start function supplies a list of prompts. For each prompt, I add the tokens to an instance property named input_tokens.
The on_llm_end function supplies a LLMResult.
1. The LLMResult, as I discovered in the source code, has a flatten() method that converts a LLMResult to a list of LLMResults with 1 generation.
2. I add the tokens to an instance property named output_tokens.

I used the callback handler when calling the custom chain:

 1from langchain.llms import Bedrock
 2from langchain.prompts import ChatPromptTemplate
 3from langchain.callbacks.base import BaseCallbackHandler
 4
 5import boto3
 6
 7
 8class AnthropicTokenCounter(BaseCallbackHandler):
 9    def __init__(self, llm):
10        self.llm = llm
11        self.input_tokens = 0
12        self.output_tokens = 0
13
14    def on_llm_start(self, serialized, prompts, **kwargs):
15        for p in prompts:
16            self.input_tokens += self.llm.get_num_tokens(p)
17
18    def on_llm_end(self, response, **kwargs):
19        results = response.flatten()
20        for r in results:
21            self.output_tokens = self.llm.get_num_tokens(r.generations[0][0].text)
22
23
24client = boto3.client("bedrock-runtime")
25
26llm = Bedrock(client=client, model_id="anthropic.claude-instant-v1")
27
28prompt = ChatPromptTemplate.from_template("tell me a joke about {subject}")
29chain = prompt | llm
30
31token_counter = AnthropicTokenCounter(llm)
32print(chain.invoke({"subject": "ai"}, config={"callbacks": [token_counter]}))
33print(token_counter.input_tokens)
34print(token_counter.output_tokens)

Output:

1Here's one: Why can't AI assistants tell jokes? Because they lack a sense of humor!
28
320

How to count Amazon Bedrock Claude tokens step-by-step guide

Thomas Taylor — Wed, 22 Nov 2023 11:50:00 -0500

Monitoring token consumption in Anthropic-based models can be straightforward and hassle-free. In fact, Anthropic offers a simple and effective method for accurately counting tokens using Python!

In this guide, I’ll show you how to count tokens for Amazon Bedrock Anthropic models.

Installing the Anthropic Bedrock Python Client

To begin, install the Amazon Bedrock Python client using pip:

1pip install anthropic-bedrock

For more information about the library, visit the technical documentation here.

Create the Amazon Bedrock connection

Instantiating the client is straightforward: use your AWS credentials to authenticate.

Since the AnthropicBedrock client uses botocore for authentication, you may use any AWS provider of your choosing:

Environment variables
AWS Configuration file
Hardcoded aws_access_key + aws_secret_key in the AnthropicBedrock client directly

Here is the code snippet from the documentation:

 1import anthropic_bedrock
 2from anthropic_bedrock import AnthropicBedrock
 3
 4client = AnthropicBedrock(
 5    # Authenticate by either providing the keys below or use the default AWS credential providers, such as
 6    # using ~/.aws/credentials or the "AWS_SECRET_ACCESS_KEY" and "AWS_ACCESS_KEY_ID" environment variables.
 7    aws_access_key="",
 8    aws_secret_key="",
 9    # Temporary credentials can be used with aws_session_token.
10    # Read more at https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html.
11    aws_session_token="",
12    # aws_region changes the aws region to which the request is made. By default, we read AWS_REGION,
13    # and if that's not present, we default to us-east-1. Note that we do not read ~/.aws/config for the region.
14    aws_region="us-east-1",
15)

For the sake of this guide, I set my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. My client is using the default region of us-east-1:

1import anthropic_bedrock
2from anthropic_bedrock import AnthropicBedrock
3
4client = AnthropicBedrock()

Listing model access

The completions Anthropic API requires a model id. To determine what models are accessible on your AWS account, run the following command using the aws cli:

1aws bedrock list-foundation-models --by-provider anthropic --query "modelSummaries[*].modelId"

Output:

1[
2    "anthropic.claude-instant-v1",
3    "anthropic.claude-v1",
4    "anthropic.claude-v2"
5]

Calling Claude on Bedrock using the client

Using the anthropic.claude-instant-v1 model id from above, here’s how to call it using the completions API:

import anthropic_bedrock
from anthropic_bedrock import AnthropicBedrock

client = AnthropicBedrock()

completion = client.completions.create(
    model="anthropic.claude-instant-v1",
    prompt=f"{anthropic_bedrock.HUMAN_PROMPT} Tell me a funny cowboy joke {anthropic_bedrock.AI_PROMPT}",
    max_tokens_to_sample=2000,
)

print(completion.completion)

Output:

1 Here's one: Why don't cowboys like to eat beef jerky in the desert? Because it's too chewy!

In the code above, I’m calling the client.completions.create function and supplying it:

The model id as a string
The prompt using the Anthropic provided constants: anthropic_bedrock.HUMAN_PROMPT and anthropic_bedrock.AI_PROMPT
The max tokens to sample is the maximum number of tokens to generate before stopping.

The prompt above resolved to this:

1import anthropic_bedrock
2
3print(
4    f"{anthropic_bedrock.HUMAN_PROMPT} Tell me a funny cowboy joke {anthropic_bedrock.AI_PROMPT}"
5)

Output:

1
2
3Human: Tell me a funny cowboy joke
4
5Assistant:

For more information about asynchronous or streamed responses, please refer to the technical documentation here

Counting tokens using the client

Using the example before, we can count the token usage using a static method provided by Anthropic.

1from anthropic_bedrock import AnthropicBedrock
2
3client = AnthropicBedrock()
4print(client.count_tokens("Hello world!"))

As of November 2023, Anthropic charges based on prompt tokens and completion tokens.

For tracking tokens, you can leverage this method to estimate:

 1import anthropic_bedrock
 2from anthropic_bedrock import AnthropicBedrock
 3
 4client = AnthropicBedrock()
 5
 6prompt = f"{anthropic_bedrock.HUMAN_PROMPT} Tell me a funny cowboy joke {anthropic_bedrock.AI_PROMPT}"
 7prompt_tokens = client.count_tokens(prompt)
 8print(prompt_tokens)
 9
10result = client.completions.create(
11    model="anthropic.claude-instant-v1",
12    prompt=prompt,
13    max_tokens_to_sample=2000,
14)
15
16completion = result.completion
17completion_tokens = client.count_tokens(completion)
18print(completion_tokens)

Output:

114
228

How to prompt DALL-E effectively

Thomas Taylor — Tue, 21 Nov 2023 00:25:00 -0500

DALL-E is an OpenAI model that generates images from textual descriptions. It combines elements of creativity and technology to produce unique imagery based on human prompts.

Tip 1 - DALL-E Prompt Book

To kick things off, I recommend reviewing the DALL-E prompt book released by the dallery gallery. It contains many useful techniques for describing tone, emotion, angles, lightning, illustration style, etc. It has been a wonderful resource for myself and will likely give you enough information to get started.

The mascot for https://how.wtf is a bald character, depicted frustratingly slamming his hands on a desk while shouting:

Tip 2 - Upload an image and prompt DALL-E

Seeking an image of the same character riding a llama, I:

Uploaded the image
Prompted DALL-E with a brief request: “Given the uploaded image, render an animated version riding a llama”

Tip 3 - Ask DALL-E to describe an image

If you wish to reproduce or modify an image, DALL-E can help describe it. For instance, I asked DALL-E to describe my mascot riding the llama:

Me: How should I prompt you to get the image attached?

DALL-E: To create an image similar to the one attached, you could use a prompt like this:

“Create an image of a joyful, cartoonish, bald character with large expressive eyes, riding on the back of a fluffy, stylized alpaca. The character has a wide, excited smile and is holding onto reins as if they are having the time of their life. The alpaca appears to be in mid-gallop, creating small clouds of dust behind it to show its speed. The environment is a soft, minimalistic background that doesn’t draw attention away from the action of the ride. The lighting should be bright and cheerful, capturing the sense of fun and adventure in the scene.”

With DALL-E’s guidance, I then requested a Simpsons-style version in a 16:9 aspect ratio:

Me: Create a 16:9 aspect ratio image of a joyful, cartoonish, bald character with large expressive eyes, riding on the back of a fluffy, stylized alpaca in the style of “The Simpsons”. The character has a wide, excited smile and is holding onto reins as if they are having the time of their life. The alpaca appears to be in mid-gallop, creating small clouds of dust behind it to show its speed. The environment is a soft, minimalistic background that doesn’t draw attention away from the action of the ride. The lighting should be bright and cheerful, capturing the sense of fun and adventure in the scene.

This resulted in the featured image for this blog post!

Conclusion

DALL-E’s remarkable ability to generate customized images from textual prompts is evident. By using specific descriptors and understanding the nuances of effective prompting, as detailed in the DALL-E prompt book, users can create vibrant, expressive images that vividly capture their ideas with striking accuracy and creativity.

OpenAI Assistants vs GPTs

Thomas Taylor — Sun, 19 Nov 2023 23:00:00 -0500

OpenAI has two products that are similar: Assistants and GPTs. In this article, we’ll discuss the differences between the two.

What are OpenAI GPTs?

GPTs (custom versions of ChatGPT) are specialized tools designed to cater to specific needs and preferences. They enable users, including those without coding skills, to tailor ChatGPT for distinct tasks or interests.

Users can create GPTs for private or public use, with the forthcoming GPT Store showcasing creations from verified builders in various categories like productivity and education. GPTs emphasize privacy and safety, ensuring user data is controlled and protected, with options for integrating third-party APIs and using chats for model improvement. This feature is particularly beneficial for enterprise customers, allowing them to develop internal-only GPTs for specific company needs. The initiative represents a step towards more interactive and capable AI systems, termed “agents,” and reflects OpenAI’s commitment to involving a broader community in AI development.

ChatGPT Plus now offers updated information and simplified access to various tools, including DALL·E, web browsing, and file attachments for enhanced user experience.

What are OpenAI Assistants?

OpenAI Assistants are AI tools integrated within applications, designed to respond to user queries with tailored instructions and capabilities. They use various models and leverage specific tools, such as Code Interpreter, Retrieval, and Function calling, to provide targeted responses and solutions.

The Assistants API enables developers to embed AI assistants into their own platforms, enhancing the functionality and user interaction experience. This API is an adaptable framework, set to expand with more OpenAI-developed tools and options for custom tool integration.

OpenAI Assistant Example

Using the OpenAI open-source Python module, we can leverage assistants easily:

WARNING: This code uses the beta OpenAI client.

 1from time import sleep
 2
 3from openai import OpenAI
 4
 5
 6def wait_for_run(run, thread):
 7    while run.status in ["queued", "in_progress"]:
 8        run = client.beta.threads.runs.retrieve(
 9            thread_id=thread.id,
10            run_id=run.id,
11        )
12        sleep(0.5)
13    return run
14
15
16client = OpenAI()
17
18assistant = client.beta.assistants.create(
19    instructions="You are a ghost writer for the famous wrapper Eminem. When asked to write an song, write in an Enimen-like rhyme scheme.",
20    model="gpt-4-1106-preview",
21)
22
23thread = client.beta.threads.create(
24    messages=[{"role": "user", "content": "Write a song about AI."}]
25)
26
27run = client.beta.threads.runs.create(thread_id=thread.id, assistant_id=assistant.id)
28
29run = wait_for_run(run, thread)
30messages = client.beta.threads.messages.list(thread.id)
31print(messages.data[0].content[0].text.value)

From a high-level, this code does the following:

Creates a thread with a prepopulated message from the user with the message “Write a song about AI.”
Runs the assistant on the thread
Polls the run for a completion status every half-second
Prints the latest message’s content

What is the difference between Assistants and GPTs?

tl;dr - Assistants are accessed through OpenAI’s API for developers. GPTs are used to personalize the ChatGPT experience.

The key difference between Assistants and GPTs is their application and integration scopes. Assistants are primarily API-driven tools meant for embedding within external applications, focusing on enhancing functionality with specific tasks like code interpretation, information retrieval, and function execution. They are components in a broader system, augmenting existing applications with AI capabilities.

On the other hand, GPTs are standalone, user-customized versions of ChatGPT, designed to fulfill varied personal, educational, or professional needs. While Assistants enhance specific functionalities within applications, GPTs are more about creating a personalized ChatGPT experience, tailored to the user’s unique requirements.

Langchain with Amazon Bedrock

Thomas Taylor — Sat, 18 Nov 2023 02:00:00 -0500

In this article, we’ll cover how to easily leverage Amazon Bedrock in Langchain.

What is Amazon Bedrock?

Amazon Bedrock is a fully managed service that provides an API to invoke LLMs. At the time of writing, Amazon Bedrock supports:

Anthropic Claude
Meta Llama
Cohere Command
Stability AI SDXL
A121 Labs Jurassic

with many more slated to come out!

How to use Amazon Bedrock with Langchain

Langchain easily integrates with Amazon Bedrock via the langchain.llms module.

Setup credentials

In the execution environment, ensure that the appropriate credentials are configured. This can be inferred from environment variables, the ~/.aws/credentials configuration file, credentials in the Amazon Bedrock client, etc.

For the sake of this tutorial, I have exported my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY pair.

1export AWS_ACCESS_KEY_ID=xxxx
2export AWS_SECRET_ACCESS_KEY=xxxx

Create a bedrock client

Using boto3, a bedrock-runtime client can be created and passed as a parameter to the Langchain Bedrock class.

1from langchain.llms import Bedrock
2
3import boto3
4
5client = boto3.client("bedrock-runtime")
6
7model = Bedrock(client=client, model_id="anthropic.claude-instant-v1")
8
9print(model("who are you?"))

Output:

1I'm Claude, an AI assistant created by Anthropic.

Creating a chain with Bedrock

Now that the model is instantiated, we may use it with the Langchain Expression Language (LCEL) to create a chain.

 1from langchain.llms import Bedrock
 2from langchain.schema.output_parser import StrOutputParser
 3from langchain.prompts import ChatPromptTemplate
 4
 5import boto3
 6
 7client = boto3.client("bedrock-runtime")
 8model = Bedrock(client=client, model_id="anthropic.claude-instant-v1")
 9prompt = ChatPromptTemplate.from_template("write me a haiku about {topic}")
10
11chain = prompt | model | StrOutputParser()
12print(chain.invoke({"topic": "money"}))

Output:

1Here is a haiku about money:
2
3Green paper bills flow
4Endless pursuit of riches
5Money rules all things

Conclusion

Integrating Langchain with Amazon Bedrock unlocks many capabilities for utilizing large language models in diverse applications. This guide has demonstrated the ease of setting up this integration, enabling you to enhance your projects with AI. As Amazon Bedrock expands its model support, the potential applications and efficiencies you can achieve will only grow.

A simple guide for hosting LLMs locally

Thomas Taylor — Fri, 17 Nov 2023 01:50:00 -0500

Hosting large langage models (LLMs) locally is simple using LocalAI.

What is LocalAI

LocalAI is an open source alternative to OpenAI. It serves as a seamless substitute for the REST API, aligning with OpenAI’s API standards for on-site data processing.

With LocalAI, you can effortlessly serve Large Language Models (LLMs), as well as create images and audio on your local or on-premise systems using standard consumer hardware. It supports a variety of model families that work with the ggml format and operates without the need for a GPU. Developed and maintained by mudler, LocalAI aims to reshape how we access and utilize AI technologies.

How to host large language models locally

Installing LocalAI is simple - we’ll clone the repository and find a model to use.

Clone the repository

1git clone https://github.com/mudler/LocalAI.git
2cd LocalAI

Select a model

Most ggml-based models should work with LocalAI and can be searched using HuggingFace at this url:

https://huggingface.co/models?search=ggml

Download the model

For this tutorial, I’m using the TheBloke/Llama-2-7B-Chat-GGML model.

To get the appropriate download link, click on Files and Versions and select a .bin file. I chose the llama-2-7b-chat.ggmlv3.q2_K.bin file.

1wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q2_K.bin \
2    -O models/Llama-2-7B-Chat

Run the docker container

Run the docker-compose command:

1docker compose up -d --pull always

This may take a few minutes since the docker image was greater than 5 GB.

Invoke the LocalAI endpoint

1curl http://localhost:8080/v1/chat/completions \
2    -H "Content-Type: application/json" \
3    -d '{"model":"Llama-2-7B-Chat","messages":[{"role":"user","content":"Who are you?"}],"temperature": 0.9}'

Output:

 1{
 2    "created": 1700201935,
 3    "object": "chat.completion",
 4    "id": "b5e4c3ff-c8ad-4350-89cb-293d2cfd806c",
 5    "model": "Llama-2-7B-Chat",
 6    "choices": [{
 7        "index": 0,
 8        "finish_reason": "stop",
 9        "message": {
10            "role": "assistant",
11            "content": "\nYou are a unique individual, created in the image of God. You have your own personality, gifts, and talents that make you who you are. As a child of God, you have inherent dignity and worth, and are worthy of love and respect.\n\nWhat is your purpose?\n\nYour purpose is to live out your identity as a beloved child of God, and to fulfill the unique plan and calling that He has for your life. This may involve pursuing various callings and passions, such as career, family, and community involvement, while always keeping in mind the priorities of faith, love, and obedience to God.\n\nWhat are some challenges you face?\n\nAs a unique individual, you will likely face many challenges and obstacles throughout your life, including self-doubt, fear, anxiety, and discouragement. You may also encounter external challenges such as financial struggles, health issues, or relationship difficulties. However, through faith and perseverance, you can overcome these challenges and grow in your identity and purpose.\n\nHow can you grow in your identity and purpose?\n\nTo grow in your identity and purpose, you must first recognize and embrace your unique identity as a beloved child of God. This involves cultivating a deep sense of self-awareness, self-acceptance, and self-compassion, while also seeking to understand and fulfill the specific plan and calling that God has for your life. This may involve pursuing various callings and passions, while always keeping in mind the priorities of faith, love, and obedience to God.\n\nWhat are some practical steps you can take?\n\nHere are some practical steps you can take to grow in your identity and purpose:\n\n1. Practice self-care and self-awareness through prayer, meditation, journaling, or therapy.\n2. Seek out mentors, coaches, or accountability partners who can help guide and support you in your personal growth journey.\n3. Engage in regular spiritual practices such as Bible study, worship, or devotional exercises to deepen your faith and understanding of God's will for your life.\n4. Set clear goals and priorities for your life, based on your unique identity and purpose, and work towards achieving them through consistent effort and persever"
12        }
13    }],
14    "usage": {
15        "prompt_tokens": 0,
16        "completion_tokens": 0,
17        "total_tokens": 0
18    }
19}

Using Langchain to call hosted model

The purpose of LocalAI is to adhere to OpenAI’s API specification. Because of this, the OpenAI Python client may be used in Langchain:

 1import os
 2
 3from langchain.llms import OpenAI
 4
 5llm = OpenAI(
 6    temperature=0.9,
 7    openai_api_key="123",
 8    openai_api_base="http://localhost:8080/v1",
 9    model_name="Llama-2-7B-Chat",
10)
11text = "What would be a great name for a dog?"
12print(llm(text))

Output:

1I'm looking for something unique and fun. I want it to be a name that stands out and is memorable. Can you help me come up with some ideas?
2
3I don't want anything too common or boring, like Max or Bella. I want something that will make my dog stand out in a crowd.
4
5Can you give me some suggestions?
6
7I would love to hear your ideas!

How to use Chroma DB step-by-step guide

Thomas Taylor — Thu, 16 Nov 2023 00:45:00 -0500

Vector databases have seen an increase in popularity due to the rise of Generative AI and Large Language Models (LLMs). Vector databases can be used in tandem with LLMs for Retrieval-augmented generation (RAG) - i.e. a framework for improving the quality of LLM responses by grounding prompts with context from external systems.

What is Chroma DB?

Chroma is an open-source embedding database that enables retrieving relevant information for LLM prompting. It emphasizes developer productivity, speed, and ease-of-use.

Chroma provides several great features:

Use in-memory mode for quick POC and querying.
Reuse collections between runs with persistent memory options.
Add and delete documents after collection creation.
Query based on document metadata & page content.

Installing Chroma DB

To install Chroma DB for Python, simply run the following pip command:

1pip install chromadb

Creating a vector database using Chroma DB

Chroma organizes data using a collection primitive type to manage collections of embeddings.

Create a collection using default embedding function

1import chromadb
2
3client = chromadb.Client()
4client.create_collection(name="collection_name")

Create a collection using specific embedding function

Chroma uses the all-MiniLM-L6-v2 model for creating embeddings. When instantiating a collection, we can provide the embedding function. Here, we’ll use the default function for simplicity.

1import chromadb
2from chromadb.utils import embedding_functions
3
4ef = embedding_functions.DefaultEmbeddingFunction()
5client = chromadb.Client()
6client.create_collection(name="collection_name", embedding_function=ef)

Chroma offers integrations with vendors such as OpenAI, Hugging Face, Cohere and more.

Writing a custom embed function is also supported:

1from chromadb import Documents, EmbeddingFunction, Embeddings
2
3class MyEmbeddingFunction(EmbeddingFunction):
4    def __call__(self, texts: Documents) -> Embeddings:
5        # embed the documents somehow
6        return embeddings

Using collections in Chroma DB

In this section, we explore useful techniques for managing and using collections in Chroma DB.

Delete collections

1import chromadb
2
3client = chromadb.Client()
4client.create_collection(name="collection_name")
5client.delete_collection(name="collection_name")

Get existing collection

1import chromadb
2
3client = chromadb.Client()
4client.create_collection(name="collection_name")
5collection = client.get_collection(name="collection_name")

Get or create collection

Chroma provides a convenient method for either getting an existing collection or creating a new one:

1import chromadb
2
3client = chromadb.Client()
4collection = client.get_or_create_collection(name="collection_name")

Adding data to collections

Adding data to a collection is straightforward with Chroma’s API. When you pass a list of documents to Chroma, it automatically tokenizes, embeds, and stores them.

1import chromadb
2
3client = chromadb.Client()
4collection = client.get_or_create_collection(name="collection_name")
5collection.add(
6    documents=["document1", "document2", "document3"],
7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
8    ids=["1", "2", "3"]
9)

The documents field provides the raw text for automatic conversion using the embedding function
metadatas is an optional field corresponding to the uploaded documents. It’s optional but recommended for later querying.
ids is a required field for the documents. The order is important as it matches the document sequence.

Querying data in a collection

 1import chromadb
 2
 3client = chromadb.Client()
 4collection = client.get_or_create_collection(name="collection_name")
 5collection.add(
 6    documents=["document1", "document2", "document3"],
 7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8    ids=["1", "2", "3"]
 9)
10
11results = collection.query(
12    query_texts=["document"],
13    n_results=2,
14    where={"type": "article"}
15)
16
17print(results)

Output:

 1{
 2    'ids': [['2', '3']],
 3    'distances': [
 4        [0.5050200819969177, 0.5763288140296936]
 5    ],
 6    'metadatas': [
 7        [{'type': 'article'}, {'type': 'article'}]
 8    ],
 9    'embeddings': None,
10    'documents': [
11        ['document2', 'document3']
12    ],
13    'uris': None,
14    'data': None
15}

The query_texts field provides the raw query string, which is automatically processed using the embedding function.
n_results specifies the number of results to retrieve.
The where clause enables metadata-based filtering.

In addition, the where field supports various operators:

 1import chromadb
 2
 3client = chromadb.Client()
 4collection = client.get_or_create_collection(name="collection_name")
 5collection.add(
 6    documents=["document1", "document2", "document3"],
 7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8    ids=["1", "2", "3"]
 9)
10
11results = collection.query(
12    query_texts=["document"],
13    n_results=2,
14    where={
15        "type": {
16            "$eq": "article"
17        }
18    }
19)

For the list of supported operators, refer to the documentation here.

Filtering based on document content is also possible:

 1import chromadb
 2
 3client = chromadb.Client()
 4collection = client.get_or_create_collection(name="collection_name")
 5collection.add(
 6    documents=["document1", "document2", "document3"],
 7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8    ids=["1", "2", "3"]
 9)
10
11results = collection.query(
12    query_texts=["document"],
13    n_results=2,
14    where={"type": "article"},
15    where_document={"$contains": "2"}
16)
17
18print(results)

Output:

1{
2    'ids': [['2']],
3    'distances': [[0.5050200819969177]],
4    'metadatas': [[{'type': 'article'}]],
5    'embeddings': None,
6    'documents': [['document2']],
7    'uris': None,
8    'data': None
9}

Getting documents based on id in the collection

Retrieve specific documents by their IDs using the collection.get method:

 1import chromadb
 2
 3client = chromadb.Client()
 4collection = client.get_or_create_collection(name="collection_name")
 5collection.add(
 6    documents=["document1", "document2", "document3"],
 7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8    ids=["1", "2", "3"]
 9)
10
11results = collection.get(
12    ids=["1"]
13)
14
15print(results)

Output:

1{
2    'ids': ['1'],
3    'embeddings': None,
4    'metadatas': [{'type': 'recipe'}],
5    'documents': ['document1'],
6    'uris': None,
7    'data': None
8}

where and where_document filters may also be used. If ids are not provided, all items matching the where and where_document conditions will be returned.

Getting all documents in the collection

Retrieve all documents in a collection:

 1import chromadb
 2
 3client = chromadb.Client()
 4collection = client.get_or_create_collection(name="collection_name")
 5collection.add(
 6    documents=["document1", "document2", "document3"],
 7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8    ids=["1", "2", "3"]
 9)
10
11results = collection.get()
12print(results)

Output:

 1{
 2    'ids': ['1', '2', '3'],
 3    'embeddings': None,
 4    'metadatas': [
 5        {'type': 'recipe'},
 6        {'type': 'article'},
 7        {'type': 'article'}
 8    ],
 9    'documents': ['document1', 'document2', 'document3'],
10    'uris': None,
11    'data': None
12}

Updating documents in a collection

Update existing documents in a collection with new embeddings or data using the collection.update method.

 1import chromadb
 2
 3client = chromadb.Client()
 4collection = client.get_or_create_collection(name="collection_name")
 5collection.add(
 6    documents=["document1", "document2", "document3"],
 7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8    ids=["1", "2", "3"]
 9)
10collection.update(
11    documents=["document10"],
12    metadatas=[{"type": "article"}],
13    ids=["1"]
14)
15
16results = collection.get(
17    ids=["1"]
18)
19print(results)

Output:

1{
2    'ids': ['1'],
3    'embeddings': None,
4    'metadatas': [{'type': 'article'}],
5    'documents': ['document10'],
6    'uris': None,
7    'data': None
8}

if a provided id is not found, an error is logged and the update is ignored.

Chroma also supports the collection.upsert operation as well:

1import chromadb
2
3client = chromadb.Client()
4collection = client.get_or_create_collection(name="collection_name")
5collection.upsert(
6    documents=["document1", "document2", "document3"],
7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
8    ids=["1", "2", "3"]
9)

Deleting documents in a collection

Documents can be deleted from a collection using the collection.delete method:

 1import chromadb
 2
 3client = chromadb.Client()
 4collection = client.get_or_create_collection(name="collection_name")
 5collection.upsert(
 6    documents=["document1", "document2", "document3"],
 7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8    ids=["1", "2", "3"]
 9)
10
11collection.delete(ids=["1"])
12print(collection.count())

Output:

Similarly to the collection.get, an optional where filter may be supplied. If no ids are supplied, it deletes all the documents that match the where conditions.

Counting the number of documents in a collection

 1import chromadb
 2
 3client = chromadb.Client()
 4collection = client.get_or_create_collection(name="collection_name")
 5collection.upsert(
 6    documents=["document1", "document2", "document3"],
 7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8    ids=["1", "2", "3"]
 9)
10
11print(collection.count())

Output:

Modifying an existing collection

Collection names may be modified after creation:

1import chromadb
2
3client = chromadb.Client()
4collection = client.get_or_create_collection(name="collection_name")
5collection.modify(name="new_collection_name")

Listing all collections in a client

1import chromadb
2
3client = chromadb.Client()
4client.get_or_create_collection(name="collection_one")
5client.get_or_create_collection(name="collection_two")
6print(client.list_collections())

Output:

1[Collection(name=collection_two), Collection(name=collection_one)]

Persisting vector databases using Chroma DB

The in-memory Chroma client provides saving and loading to disk functionality with the PersistentClient.

Saving to disk

1import chromadb
2
3client = chromadb.PersistentClient(path="document_store")
4collection = client.get_or_create_collection(name="collection_name")
5collection.upsert(
6    documents=["document1", "document2", "document3"],
7    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
8    ids=["1", "2", "3"]
9)

Loading an existing from disk

 1import chromadb
 2
 3client = chromadb.PersistentClient(path="document_store")
 4collection = client.get_or_create_collection(name="collection_name")
 5#collection.upsert(
 6#    documents=["document1", "document2", "document3"],
 7#    metadatas=[{"type": "recipe"}, {"type": "article"}, {"type": "article"}],
 8#    ids=["1", "2", "3"]
 9#)
10print(collection.get())

Output:

{
    'ids': ['1', '2', '3'],
    'embeddings': None,
    'metadatas': [
        {'type': 'recipe'},
        {'type': 'article'},
        {'type': 'article'}
    ],
    'documents': ['document1', 'document2', 'document3'],
    'uris': None,
    'data': None
}

Running Chroma DB as a server

Chroma may be used as a standalone server. Simply run the following command:

1chroma run --path /document_store

Output:

 1
 2
 3                (((((((((    (((((####
 4             ((((((((((((((((((((((#########
 5           ((((((((((((((((((((((((###########
 6         ((((((((((((((((((((((((((############
 7        (((((((((((((((((((((((((((#############
 8        (((((((((((((((((((((((((((#############
 9         (((((((((((((((((((((((((##############
10         ((((((((((((((((((((((((##############
11           (((((((((((((((((((((#############
12             ((((((((((((((((##############
13                (((((((((    #########
14
15
16
17Running Chroma
18
19Saving data to: document_store
20Connect to chroma at: http://localhost:8000
21Getting started guide: https://docs.trychroma.com/getting-started

Then connect to the local host using the HttpClient:

1import chromadb
2
3client = chromadb.HttpClient(host="localhost", port=8000)
4print(client.list_collections())

Output:

1[Collection(name=collection_name)]

Differences between Langchain and LlamaIndex and their use cases

Thomas Taylor — Tue, 14 Nov 2023 00:20:00 -0500

What are the differences between Langchain and LlamaIndex and what are their use cases? In this article, we’ll explore these differences, helping you choose the most suitable library for your needs.

What is Langchain?

As the documentation outlines,

LangChain is a framework for developing applications powered by language models.

Langchain is a framework that enables developers to build solutions powered by language models.

Some use-case examples include:

Retrieval-augmented generation (RAG)
Interacting with APIs
Chatbots
Extraction
Summarization
Data Generation

The main value additions from Langchain include:

Components: modular tooling that is easy-to-use whether or not you’re using the rest of Langchain’s framework
Off-the-shelf-chains: components assembled together to accomplish a higher-level task

In addition, Langchain has made strides to streamline hosting and deploying chains with tools like LangSmith and LangServe.

What is LlamaIndex?

LlamaIndex is a data framework specifically designed for language model solutions, focusing on data ingestion and querying. Its primary aim is to streamline the retrieval of necessary information, providing the language model with sufficient context to deliver the desired results.

At its core, the data framework provides a set of tooling that streamlines data retrieval:

Data connectors: ingesting data from existing data stores
Data indexes: transform the data for the LLMs to easily consume
Engines: a purposed method for interacting with data: chat engine, query engines for RAG, etc.
Data Agents: LLM-powered workers that work with tools to access domain data

Which should I use?

Choosing between the two isn’t necessarily an either/or decision, as LlamaIndex can integrate with Langchain. You can’t go wrong leveraging Langchain and using LlamaIndex for the RAG-specific scenarios since it’s optimized for it.

If you prefer a more streamlined RAG solution without the additional features of Langchain, LlamaIndex is an excellent standalone choice.

Reverting changes to Git submodules

Thomas Taylor — Mon, 13 Nov 2023 00:10:00 -0500

Reverting a submodule in Git is simple.

How to revert changes to a Git submodule

The cleanest / easiest method of reverting changes to one or many git submodules is by using the following two commands:

1git submodule deinit -f .
2git submodule update --init

The git submodule deinit -f . command “deinitializes” all submodules in the repository.

The git submodule update --init command does a new checkout of them.

Create Lambda Function URL using AWS CDK

Thomas Taylor — Sun, 12 Nov 2023 01:50:00 -0500

Deploying Lambda Function URLs with the AWS CDK is simple.

How to deploy Function URLs using AWS Python CDK

Follow the instructions below to deploy an AWS Lambda Function with a URL.

Create a requirements.txt

For this tutorial, version 2.106.1 of the AWS CDK was used.

1aws-cdk-lib==2.106.1
2constructs>=10.0.0,<11.0.0

Install the dependencies

1pip3 install -r requirements.txt

Create /handler folder with an index.py file

1def handler(event, context):
2    return {
3        "statusCode": 200,
4        "body": {"message": "👋"}
5    }

Create stack.py with a basic lambda function

 1from os import path
 2
 3import aws_cdk as cdk
 4import aws_cdk.aws_lambda as lambda_
 5
 6
 7class FunctionUrlStack(cdk.Stack):
 8    def __init__(self, scope: cdk.App, construct_id: str, **kwargs) -> None:
 9        super().__init__(scope, construct_id, **kwargs)
10
11        lambda_function = lambda_.Function(
12            self,
13            "Function",
14            runtime=lambda_.Runtime.PYTHON_3_11,
15            handler="index.handler",
16            code=lambda_.Code.from_asset(path.join(path.dirname(__file__), "handler")),
17        )
18
19        function_url = lambda_function.add_function_url(
20            auth_type=lambda_.FunctionUrlAuthType.NONE
21        )
22
23        cdk.CfnOutput(self, "FunctionUrl", value=function_url.url)

Create app.py

1import aws_cdk as cdk
2
3from stack import FunctionUrlStack
4
5app = cdk.App()
6FunctionUrlStack(app, "FunctionUrlStack")
7
8app.synth()

Create cdk.json

1{
2  "app": "python3 app.py"
3}

The directory structure should look like this:

1project/
2├── app.py
3├── cdk.json
4├── handler
5│   └── index.py
6├── requirements.txt
7└── stack.py

Deploy the stack

1cdk deploy

Because of the CfnOutput, the distribution’s domain name is in an output:

1Outputs:
2FunctionUrlStack.FunctionUrl = https://.lambda-url.us-east-1.on.aws/

The link will display the following text:

1{
2    "message": "👋"
3}

Custom memory in Langchain

Thomas Taylor — Sat, 11 Nov 2023 22:10:00 -0500

Implementing custom memory in Langchain is dead simple using the ChatMessageHistory class.

How to implement custom memory in Langchain (including LCEL)

One of the easiest methods for storing and retrieving messages with Langchain is using the ChatMessageHistory class that is provided from the langchain.memory module.

It’s simple to get started with:

 1from langchain.memory import ChatMessageHistory
 2
 3history = ChatMessageHistory()
 4
 5history.add_user_message("Hello!")
 6
 7history.add_ai_message("Yo!")
 8
 9print(history.messages)
10
11# [HumanMessage(content='Hello!'), AIMessage(content='Yo!')]

The interface for ChatMessageHistory is: add_user_message and add_ai_message.

For the sake of this example, I’m using a .json file with preloaded messages. A database, file, etc. may be used instead.

 1{
 2    "messages": [
 3        {
 4            "sender": "human",
 5            "body": "Hello!"
 6        },
 7        {
 8            "sender": "ai",
 9            "body": "Yo!"
10        }
11    ]
12}

 1import json
 2
 3from langchain.memory import ChatMessageHistory
 4
 5with open("messages.json") as json_file:
 6    content = json.load(json_file)
 7
 8messages = content["messages"]
 9
10chat_history = ChatMessageHistory()
11for m in messages:
12    if m["sender"] == "human":
13        chat_history.add_user_message(m["body"])
14    elif m["sender"] == "ai":
15        chat_history.add_ai_message(m["body"])
16
17print(chat_history.messages)
18# [HumanMessage(content='Hello!'), AIMessage(content='Yo!')]

The chat_history may be used for instantiating other types of memory!

Here is an example from the langchain documentation using the ChatMessageHistory with Langchain Expression Language (LCEL):

 1import json
 2
 3from operator import itemgetter
 4
 5from langchain.chat_models import ChatOpenAI
 6from langchain.memory import ConversationBufferMemory
 7from langchain.schema.runnable import RunnablePassthrough, RunnableLambda
 8from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
 9
10with open("messages.json") as json_file:
11    content = json.load(json_file)
12
13messages = content["messages"]
14
15chat_history = ChatMessageHistory()
16for m in messages:
17    if m["sender"] == "human":
18        chat_history.add_user_message(m["body"])
19    elif m["sender"] == "ai":
20        chat_history.add_ai_message(m["body"])
21
22model = ChatOpenAI()
23prompt = ChatPromptTemplate.from_messages(
24    [
25        ("system", "You are a helpful chatbot"),
26        MessagesPlaceholder(variable_name="history"),
27        ("human", "{input}"),
28    ]
29)
30
31memory = ConversationBufferMemory(return_messages=True, chat_memory=chat_history)
32
33chain = (
34    RunnablePassthrough.assign(
35        history=RunnableLambda(memory.load_memory_variables) | itemgetter("history")
36    )
37    | prompt
38    | model
39)
40
41inputs = {"input": "hi im bob"}
42response = chain.invoke(inputs)

How to iterate through JSON arrays in Bash using jq

Thomas Taylor — Tue, 13 Jun 2023 22:45:00 -0400

Learn the simple process of iterating through a JSON array in bash with the help of jq.

Iterate through a JSON array using `jq`

Scenario: You have a file named projects.json contains an array named “projects” that you want to iterate through.

 1{
 2	"projects": [
 3		{
 4			"name": "project 1",
 5			"owner": "owner 1",
 6			"id": 1,
 7			"version": "2.3.0"
 8		},
 9		{
10			"name": "project 2",
11			"owner": "owner 1",
12			"id": 2,
13			"version": "1.54.0"
14		},
15		{
16			"name": "project 3",
17			"owner": "owner 2",
18			"id": 3,
19			"version": "4.9.2"
20		}
21	]
22}

To achieve this, you can use the bash snippet below:

1jq -c '.projects[]' projects.json | while read i; do
2    echo "$i"
3done

The command uses jq to extract each object in the “projects” array and then uses a while loop to iterate through the extracted objects. The echo statement prints each object.

The output will be:

1{"name":"project 1","owner":"owner 1","id":1,"version":"2.3.0"}
2{"name":"project 2","owner":"owner 1","id":2,"version":"1.54.0"}
3{"name":"project 3","owner":"owner 2","id":3,"version":"4.9.2"}

Iterate through a JSON array using `jq` and associative arrays in Bash

If your version of bash supports associative arrays, you can use an alternative approach:

1readarray -t projects < <(jq -c '.projects[]' projects.json)
2for i in "${projects[@]}"; do
3    echo "$i"
4done

This command stores each object from the “projects” array in an associative array called projects. Then, it uses a for loop to iterate through the objects and echo to print each object.

The output will be the same as before:

1{"name":"project 1","owner":"owner 1","id":1,"version":"2.3.0"}
2{"name":"project 2","owner":"owner 1","id":2,"version":"1.54.0"}
3{"name":"project 3","owner":"owner 2","id":3,"version":"4.9.2"}

DynamoDB auto incrementing keys: Why UUIDs are the recommended choice

Thomas Taylor — Sun, 11 Jun 2023 11:05:00 -0400

When working with Amazon DynamoDB, the design of the primary key plays a crucial role. Unlike traditional relational databases, DynamoDB does not natively support auto-incrementing primary keys. However, there are alternative approaches that provide greater flexibility and scalability.

In this article, we will explore the drawbacks of auto-incrementing primary keys in DynamoDB and present UUIDs as a recommended alternative.

The Problem with Auto-Incrementing Keys

Auto-incrementing keys can result in hotspots and performance bottlenecks in DynamoDB. This occurs because new records are appended to the end of the table or partition, leading to an uneven distribution of data.

Furthermore, this uneven distribution can cause certain partitions to be overloaded while others remain underutilized, adversely impacting the scalability and performance of the system.

Alternatives to Auto-Incrementing

As a viable alternative, UUIDs (Universally Unique IDs) offer a solution.

UUIDs ensure a low probability of collisions, even when generated across distributed systems. This makes them highly suitable for maintaining uniqueness in DynamoDB primary keys.

 1import boto3
 2import uuid
 3
 4dynamodb = boto3.resource('dynamodb')
 5table = dynamodb.Table('users')
 6table.put_item(Item={
 7    'user_id': str(uuid.uuid4()),
 8    'name': 'Foo bar',
 9    'email': 'foobar@example.com',
10    'age': 55 
11})

In the code snippet above, a user named Foo bar is inserted into a DynamoDB table using a generated UUID as the primary key.

Pico vs Nano A Comparison of Lightweight Text Editors

Thomas Taylor — Sat, 10 Jun 2023 11:15:00 -0400

When it comes to lightweight text editors, Pico and Nano are viable options that are available on Unix-based systems. While they share similarities, understanding their differences will enable users to make an informed choice.

What is Pico?

Pico (Pine composer) is a user-friendly text editor known for its simplicity. It features a straightforward interface with an accessible on-screen menu.

The intentions of the software are clear:

Move the cursor by using arrow keys
Insert a character by typing
Save, spell check, etc. by leveraging the on-screen menu instructions

History of Pico

Pico was integrated with the Pine email client, which was developed by the University of Washington and made public in 1992.

NOTE: Pico is not actively developed.

What is Nano?

Nano is a modern replacement for Pico that offers additional features and ongoing development. It aims to provide a more powerful editing experience over its predecessor while retaining simplicity.

Nano supports the following features and more:

Syntax highlighting for software
Multiple buffers
Regular expression matching
Customizable keybindings

History of Nano

Nano was released by Chris Allegretta in 1999 and became apart of the GNU Project in 2001. It was developed because Pico’s earlier license had ambiguous redistribution terms; however, newer version of Pico were released under Apache License V2.

Nano development continues to this day and has an active community of users.

Should `pico` or `nano` be used?

The choice between the two ultimately depends on the user’s needs and preferences.

A few notable points:

pico is not actively developed
pico does not contain common program tooling: syntax highlighting, regex search, etc.
pico does not support multiple buffers

Use virtualenv with a Python Jupyter notebook

Thomas Taylor — Thu, 08 Jun 2023 23:50:00 -0400

Using a virtual environment with a Jupyter notebook is easy!

Install and use `virtualenv`

For instructions on installing, using, and activating virtualenv, refer to my blog article here.

NOTE: You may use your virtual environment of choice.

Add `virtualenv` environment to Jupyter notebook

Activate the virtual environment
Install ipykernel

1pip3 install ipykernel

Use the ipykernel install command

1python3 -m ipykernel install --user --name=venv # this name can be anything

Open jupyter-lab and click on the environment labeled venv

The ipykernel install command uses the current environment when creating the kernel.

Create an HTTP server in one line

Thomas Taylor — Wed, 07 Jun 2023 20:40:00 -0400

If you need a quick one-liner HTTP server from the command line, Python comes in handy.

How to create an HTTP server in one line

Python enables users to run a simple command to spin up an HTTP server:

1python -m http.server

Example:

1python -m http.server 4000

By default, this command will serve the files in the current directory using a single-threaded HTTP server bound to localhost.

In addition, users may specify a different directory:

1python -m http.server  --directory

shuf command in Linux

Thomas Taylor — Mon, 05 Jun 2023 23:55:00 -0400

What is the shuf command in Linux?

What is the `shuf` command?

The shuf command in Linux is a utility for shuffling lines of input files. It allows users to shuffle contents of a file or produce random output based on supplied arguments.

How is the `shuf` command used?

Given a file named quotes.txt with the following contents:

1"The only way to do great work is to love what you do." - Steve Jobs
2"In the end, it's not the years in your life that count. It's the life in your years." - Abraham Lincoln
3"Success is not final, failure is not fatal: It is the courage to continue that counts." - Winston Churchill
4"Believe you can and you're halfway there." - Theodore Roosevelt
5"Be yourself; everyone else is already taken." - Oscar Wilde

the shuf command allows for a random permutation of all the lines.

1% shuf quotes.txt
2"Success is not final, failure is not fatal: It is the courage to continue that counts." - Winston Churchill
3"In the end, it's not the years in your life that count. It's the life in your years." - Abraham Lincoln
4"The only way to do great work is to love what you do." - Steve Jobs
5"Believe you can and you're halfway there." - Theodore Roosevelt
6"Be yourself; everyone else is already taken." - Oscar Wilde

In addition, it can be used in the following scenarios as well:

List 1 random quote:

1% shuf -n 1 quotes.txt
2"In the end, it's not the years in your life that count. It's the life in your years." - Abraham Lincoln

List random numbers between a range of numbers:

1% shuf -i 0-5
24
31
40
52
63
75

Give shuf a list of options using -e and outputing 1 chosen line using -n:

1% shuf -e option1 option2 option3 -n 1
2option3

Standard input works as well:

1% seq 5 | shuf
25
32
44
51
63

rev command in Linux and how to reverse strings

Thomas Taylor — Sun, 04 Jun 2023 23:00:00 -0400

What is the rev command in Linux?

What is the `rev` command?

As the name implies, the rev command (short for reverse) allows users to reverse a line of characters or a file.

How is the `rev` command used?

The rev command is typically supplied a string of characters or a file input.

`rev` with a file

If a file named example.txt contains:

1This is a line of text
2txet fo enil a si siht
3another one

and rev is supplied example.txt, then the output will showcase the reversed result.

1% rev example.txt
2txet fo enil a si sihT
3this is a line of text
4eno rehtona

`rev` with standard input

rev accepts standard input by default.

1% rev
2this is a line
3enil a si siht

1% echo this is a line | rev
2enil a si siht

This is useful for scripting capabilities:

1foo="I want this text reversed"
2bar=$(echo $foo | rev)
3echo $bar

Undo and redo changes in Vim

Thomas Taylor — Sat, 03 Jun 2023 23:25:00 -0400

Undo and redo operations are seamless using Vim.

How to undo changes in Vim

To undo changes in Vim, press u. In addition, using any combination of the :u[ndo] command:

:u
:un
:und
:undo

How to redo changes in Vim

To redo changes in Vim, press CTRL-R. In addition, using any combination :red[o] command:

:red
:redo

Invalid constructor name error in Apex

Thomas Taylor — Sat, 03 Jun 2023 12:45:00 -0400

When deploying apex classes, a user may run into an odd error: Error: Compile Error: Invalid constructor name.

How to solve `Invalid constructor name`

Upon initial inspection, this error may be misleading. The class in question contains a working constructor or a new method was added that is not a constructor.

What gives?

Normally, this error indicates that a new method was added without adding the return type.

Example:

 1public class Foo {
 2
 3    private String bar;
 4
 5    public Foo(String bar) {
 6        this.bar = bar;
 7    }
 8
 9    public newMethod() {
10        // logic
11    }
12}

To fix this, simply add the return type to the method:

 1public class Foo {
 2
 3    private String bar;
 4
 5    public Foo(String bar) {
 6        this.bar = bar;
 7    }
 8
 9    // added void return type
10    public void newMethod() {
11        // logic
12    }
13}

Why does this error happen?

When a return type is omitted, the method is interpreted as a constructor because constructors do not contain return types.

What is the apropos command in Linux

Thomas Taylor — Thu, 01 Jun 2023 23:40:00 -0400

Recently, I was made aware of the apropos command in Linux. What is it? What is it used for?

What is the `apropos` command?

apropos enables users to search terminal commands based on a keywords or descriptions.

This is an incredibly useful tool because it quickly identifies commands based on man pages.

Scenario: I forgot the command name for the qalculate CLI I installed.

1% apropos calculator
2qalc(1)                  - Powerful and easy to use command line calculator
3transicc(1)              - little cms ColorSpace conversion calculator
4bc(1)                    - arbitrary-precision decimal arithmetic language and calculator
5dc(1)                    - arbitrary-precision decimal reverse-Polish notation calculator
6bc(1)                    - arbitrary-precision decimal arithmetic language and calculator
7dc(1)                    - arbitrary-precision decimal reverse-Polish notation calculator

qalc was the first result.

Nothing appropriate error

If the man database is not updated, apropos may report an error when looking up commands:

1% apropos calculator
2calculator: nothing appropriate.

To fix it, update the man pages:

1mandb

What is double asterisk kwargs in Python

Thomas Taylor — Wed, 31 May 2023 23:35:00 -0400

When developing in Python, it’s common to come across **kwargs.

What is the double asterisk (**) in Python?

If a function parameter explictly denotes the double asterisk:

1def foo(**kwargs):
2    print(kwargs)
3
4foo(bar="test", baz=5)

kwargs will represent a dictionary of the key word arguments given to the function.

NOTE: kwargs is a standard name that refers to “key word arguments”; however, any name may be used.

Output:

1{'bar': 'test', 'baz': 5}

Sort a dictionary by key or value in Python

Thomas Taylor — Tue, 30 May 2023 23:35:00 -0400

Sorting based on a dictionary value or key is easy using Python.

Sort by value in a dictionary

In Python 3.7 and above, a combination of dictionary comprehension and sorted can be used:

1foo = {"first": 1, "third": 3, "second": 2, "zeroth": 0}
2print({k: v for k, v in sorted(foo.items(), key=lambda i: i[1])})

In addition,

1foo = {"first": 1, "third": 3, "second": 2, "zeroth": 0}
2print(dict(sorted(foo.items(), key=lambda i: i[1])))

Output:

1{'zeroth': 0, 'first': 1, 'second': 2, 'third': 3}

sorted allows a optional keyword argument of key: a function to execute for ordering.

Sort by key in a dictionary

Similar to the technique before, the lambda argument can reference the 0th index to sort by the dictionary keys.

1foo = {"first": 1, "third": 3, "second": 2, "zeroth": 0}
2print({k: v for k, v in sorted(foo.items(), key=lambda i: i[0])})

Output:

1{'first': 1, 'second': 2, 'third': 3, 'zeroth': 0}

Iterate through two lists in parallel in Python

Thomas Taylor — Mon, 29 May 2023 21:30:00 -0400

Iterating through a two lists in parallel is natively handled in Python using the zip function.

Iterate through two lists using `zip`

Python’s zip function will aggregate elements from two or more iterables.

1foo = ["x", "y", "z"]
2bar = [1, 2, 3]
3
4for f, b in zip(foo, bar):
5  print(f, b)

Output:

1x 1
2y 2
3z 3

This works with any number of iterables:

1foo = ["x", "y", "z"]
2bar = [1, 2, 3]
3baz = ["!", "@", "#"]
4
5for f, b, z in zip(foo, bar, baz):
6  print(f, b, z)

Output:

1x 1 !
2y 2 @
3z 3 #

If the index is necessary, use enumerate:

1foo = ["x", "y", "z"]
2bar = [1, 2, 3]
3
4for i, (f, b) in enumerate(zip(foo, bar)):
5  print(i, f, b)

Output:

10 x 1
21 y 2
32 z 3

How to install Neovim on Ubuntu

Thomas Taylor — Sun, 28 May 2023 23:45:00 -0400

Installing Neovim on Debian-based distributions: Ubuntu, Mint, etc. is easy!

Install Neovim on Ubuntu

Neovim is available through the Debian package manager. Simply execute the following command:

1sudo apt install neovim

Install Neovim using PPA

The default Neovim version bundled with apt is normally outdated. To install a newer version, a PPA (Personal Package Archive) must be used.

NOTE: The Neovim PPAs are not maintained by the Neovim team

Neovim stable PPA

To install the latest stable Neovim version using apt, add the following PPA:

1sudo add-apt-repository ppa:neovim-ppa/stable
2sudo apt-get update
3sudo apt-get install neovim

Neovim unstable PPA

To install the latest unstable Neovim verson using apt, add the following PPA:

1sudo add-apt-repository ppa:neovim-ppa/unstable
2sudo apt-get update
3sudo apt-get install neovim

Find devices on a network in Linux

Thomas Taylor — Sat, 27 May 2023 22:55:00 -0400

Finding a device that’s connected to a network on Linux is easy with an open source tool named nmap.

What is `nmap`?

nmap is a utility for network discovery and security auditing. It’s designed to rapidly scan large networks.

Install `nmap`

nmap is available through most popular package managers.

For Debian-based distributions:

1sudo apt install nmap

For MacOS:

1brew install nmap

Using `nmap`

Firstly, grab your IP address using ifconfig

1ifconfig

the output will contain an IP.

1inet 192.168.1.91 netmask 0xffffff00 broadcast 192.168.1.255

nmap is simple to use - simply run the command like the example below:

1sudo nmap -sn 192.168.1.0/24

The /24 is CIDR notation. It’s instructing nmap to scan all ip addresses between 192.168.1.0 and 192.168.1.255.

The -sn option is a portless scan of the IP address range.

Output:

1Nmap scan report for 192.168.1.20
2Host is up (0.012s latency).
3MAC Address: 00:11:32:C2:7B:96 (Synology Incorporated)

In my case, I was looking for the IP address of my Synology NAS.

Merge JSON objects using jq

Thomas Taylor — Fri, 26 May 2023 23:40:00 -0400

Merging objects in jq is handled natively since 1.14.

Merge objects using `jq`

Given a file named example.json

1{
2    "key": "value1",
3    "anotherKey": "value2",
4    "oneMoreKey": "value3"
5}

and another file named example2.json,

1{
2  "key1": "value2",
3  "key2": "value3",
4  "key4": "value4"
5}

merging the two objects can be completed using:

1jq -s '.[0] * .[1]' example.json example2.json

Output:

1{
2  "key1": "value2",
3  "key2": "value3",
4  "key4": "value4",
5  "key": "value1",
6  "anotherKey": "value2",
7  "oneMoreKey": "value3"
8}

Get key names from JSON using jq

Thomas Taylor — Wed, 24 May 2023 23:55:00 -0400

Fetching the key names from a JSON document using jq is simple!

Get key names of JSON using `jq`

Given a file named example.json,

1{
2    "key": "value1",
3    "anotherKey": "value2",
4    "oneMoreKey": "value3"
5}

extracting the keys in alphabetical order can be completed using:

1jq 'keys' example.json

Output:

1[
2  "anotherKey",
3  "key",
4  "oneMoreKey"
5]

Get key names of JSON unsorted using `jq`

Taking the previous JSON document example.json, the keys may be returned in order as they appear:

1jq 'keys_unsorted' example.json

Output:

1[
2  "key",
3  "anotherKey",
4  "oneMoreKey"
5]

Get key names of JSON in array of objects using `jq`

Given a file named example.json,

1[
2    {
3        "key1": "value1"
4    },
5    {
6        "key2": "value2"
7    }
8]

extracting the keys from the nested array of objects can be completed using:

1jq '.[] | keys' example.json

Output:

1[
2  "key1"
3]
4[
5  "key2"
6]

If the list is required without the brackets:

1jq '.[] | keys[]' example.json

Output:

1"key1"
2"key2"

Removing the quotes is also an option:

1jq -r '.[] | keys[]' example.json

Output:

1key1
2key2

Uninstall AWS CLI

Thomas Taylor — Tue, 23 May 2023 23:55:00 -0400

The AWS CLI can be installed using several methods: this article attempts to address uninstalling through many of them.

Uninstall AWS CLI using Pip

1pip3 uninstall awscli -y
2# OR
3pip uninstall awscli -y

Uninstall AWS CLI on Ubuntu

1sudo apt-get remove --auto-remove awscli

Uninstall AWS CLI using Brew on MacOS

1brew uninstall awscli

Uninstall AWS CLI using Snap

1sudo snap rm -r aws-cli

Uninstall AWS CLI with Yum

1sudo yum erase awscli

Uninstall AWS CLI manually

If all else fails, uninstalling manually is a remaining option. Determine the executing path for the AWS CLI:

1which aws

Then remove:

1sudo rm -rf "path from previous command"

Full scan of DynamoDB table in Python

Thomas Taylor — Mon, 22 May 2023 21:35:00 -0400

A full scan of DynamoDB in Python is possible by using the LastEvaluatedKey.

Scanning DynamoDB using `LastEvaluatedKey`

According to the boto3 scan documentation,

If LastEvaluatedKey is present in the response, you need to paginate the result set. For more information, see Paginating the Results in the Amazon DynamoDB Developer Guide.

Simply check for the LastEvaluatedKey in the response using response.get('LastEvaluatedKey').

 1import boto3
 2
 3dynamodb = boto3.resource(
 4    'dynamodb',
 5    aws_session_token=token,
 6    aws_access_key_id=access_key_id,
 7    aws_secret_access_key=secret_access_key,
 8    region_name=region
 9)
10
11table = dynamodb.Table('table-name')
12
13response = table.scan()
14data = response['Items']
15while response.get('LastEvaluatedKey') in response:
16    response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
17    data.extend(response['Items'])

In the example above, the list of items (data) is extended with each page of items.

Python switch statement example

Thomas Taylor — Sun, 21 May 2023 23:55:00 -0400

Python 3.10 added support for the switch keyword.

Implement switch statements in Python

The basic syntax for a switch statement is:

1match term:
2    case pattern1:
3        some_action
4    case pattern2:
5        some_other_action
6    case _:
7        default_action

A direct example:

 1pizza_size = 0
 2match pizza_size:
 3    case 0:
 4        print("small pizza")
 5    case 1:
 6        print("medium pizza")
 7    case 2:
 8        print("large pizza")
 9    case _:
10        print("pizza size is not valid")

Output:

1small pizza

In Python, break statements are not required.

Command line recycle bin for Linux

Thomas Taylor — Sat, 20 May 2023 21:50:00 -0400

rm is a great tool that gives users enough power to inadvertently erase data. While the data may be recoverable, there is a tool that gives users recycle bin behavior.

What is `trash-cli`

trash-cli is a Python tool that implements the freedesktop.org trash specification. In contrast to rm, users can recover deleted files easily.

Installing `trash-cli`

The recommended way to install trash-cli is through pip (the Python package manager):

1pip3 install trash-cli

Alternatively, trash-cli is available on some package managers:

Debian/Ubuntu:

1sudo apt install trash-cli

Arch:

1pacman -S trash-cli

Fedora:

1dnf install trash-cli

`trash-cli` simple usage

Using trash-cli is straightforward.

Trash a file:

1trash path/to/file

Trash many files:

1trash path/to/file1 path/to/file2

List trashed files:

1$ trash-list
22023-05-20 21:43:02 /Users/thomas/example2.txt
32023-05-20 21:43:02 /Users/thomas/example1.txt

Remove specific trashed files:

1$ trash-rm example2.txt
2$ trash-list
32023-05-20 21:43:02 /Users/thomas/example1.txt

Restore trash:

1$ trash-restore
2   0 2023-05-20 21:43:02 /Users/thomas/example1.txt
3What file to restore [0..0]: 0

Lastly, emptying trash:

1$ trash-empty
2Would empty the following trash directories:
3    - /Users/thomas/.local/share/Trash
4Proceed? (y/n) y

Check if a variable is set in Bash

Thomas Taylor — Fri, 19 May 2023 22:45:00 -0400

Checking if a variable is set is easy in Bash; however, there are two common scenarios:

Checking if a variable is empty or unset
Checking if a variable is unset

Checking if a variable is empty or unset

Firstly, a simple test may be used to determine if a variable’s value is empty or if it’s unset.

1if [[ -z "$var" ]]; then
2    echo "it's empty or unset"
3fi

1var=""
2if [[ -z "$var" ]]; then
3    echo "it's empty or unset"
4fi

Output:

1it's empty or unset

The -z tests if the string length is zero.

Checking if a variable is unset

Checking if a variable is unset requires a different implementation.

1if [[ -z "${var+set}" ]]; then
2    echo "it's unset"
3fi

Output:

1it's unset

However, if the var value is set to an empty value:

1var=
2if [[ -z "${var+set}" ]]; then
3    echo "it's unset"
4fi

1var=""
2if [[ -z "${var+set}" ]]; then
3    echo "it's unset"
4fi

the output will not print anything.

The ${var+set} works because of parameter expansion:

If parameter is unset or null, null shall be substituted; otherwise, the expansion of word (or an empty string if word is omitted) shall be substituted.

Count number of files in Linux

Thomas Taylor — Thu, 18 May 2023 22:20:00 -0400

Counting the number of files in Linux is made possible using the wc command!

Count number of files and directories in current directory

Using wc in combination with ls, the number of files and directories in the current directory can be displayed:

1ls | wc -l

Output:

1~/how.wtf/content/post$ ls | wc -l
2120

To include hidden files and directories, use the -A ls option.

1ls -A | wc -l

Count number of files in current directory

Using find and wc, the number of files in the current directory can be displayed:

1find . -maxdepth 1 -type f | wc -l

-maxdepth ensures that subdirectories are not counted.

Count number of files in current directory and subdirectories

Using find and wc, the number of files excluding directories can be displayed:

1find . -type f | wc -l

Great alternatives to man pages everyone needs

Thomas Taylor — Wed, 17 May 2023 20:30:00 -0400

man (manual documentation pages) cannot be replaced; however, there are modern CLI tools that provide simplified experiences.

`tldr` pages

tldr is a community-curated collection of simplified man pages. As the name (too long; didn’t read) implies, the focus is providing straightforward examples that demonstrate tool usage.

Installing `tldr`

On MacOS,

1brew install tldr

Using the NodeJS client,

1npm -g tldr

Using the Python client,

1pip3 install tldr

Using `tldr

Using tldr is straightforward:

1tldr command name>

 1$ tldr tldr
 2
 3tldr
 4
 5Display simple help pages for command-line tools from the tldr-pages project.
 6More information: .
 7
 8- Print the tldr page for a specific command (hint: this is how you got here!):
 9    tldr command
10
11- Print the tldr page for a specific subcommand:
12    tldr command-subcommand
13
14- Print the tldr page for a command for a specific [p]latform:
15    tldr -p android|linux|osx|sunos|windows command
16
17- [u]pdate the local cache of tldr pages:
18    tldr -u

`cheat`

cheat is a tool used exclusively for supplying command usage. The intention is to provide a “cheatsheet” for *nix system administrators to quickly view command usage and contribute their own.

Installing `cheat`

On MacOS,

1brew install cheat

Using GO,

1go install github.com/cheat/cheat/cmd/cheat@latest

To find more installation examples, visit the document here.

Using `cheat`

Using cheat is simple:

1cheat command name>

 1$ cheat jq
 2# To pretty print the json:
 3jq "." < filename.json
 4
 5# To access the value at key "foo":
 6jq '.foo'
 7
 8# To access first list item:
 9jq '.[0]'
10
11# to slice and dice:
12jq '.[2:4]'
13jq '.[:3]'
14jq '.[-2:]'
15
16# to extract all keys from json:
17jq keys
18
19# to sort by a key:
20jq '.foo | sort_by(.bar)'
21
22# to count elements:
23jq '.foo | length'
24
25# print only selected fields:
26jq '.foo[] | {field_1,field_2}'
27
28# print selected fields as text instead of json:
29jq '.foo[] | {field_1,field_2} | join(" ")'
30
31# only print records where given field matches a value
32jq '.foo[] | select(.field_1 == "value_1")'

AWS SigV4 requests with curl

Thomas Taylor — Tue, 16 May 2023 22:35:00 -0400

Curl handles AWS Signature Version 4 API requests natively.

How to create AWS Signature Version 4 requests using `curl`

If an API Gateway is configured to use AWS IAM authorization, curl provides a seamless integration for HTTP requests.

1curl "$url" \
2	--user "$AWS_ACCESS_KEY_ID":"$AWS_SECRET_ACCESS_KEY" \
3	--aws-sigv4 "aws:amz:us-east-1:execute-api"

In the example, the $url links to a custom domain that points to an API Gateway.

The --user argument is given the $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY that links to the AWS IAM user.

The AWS service for invoking an API Gateway is execute-api. For curl, the full provider string is required:

1--aws-sigv4 "aws:amz:region:service"

How to use session token with AWS SigV4 curl

In addition to using IAM user credentials, you can optionally specify a session token using the header of x-amz-security-token. This is useful for temporary access through role assumptions.

1curl "$url" \
2	--user "$AWS_ACCESS_KEY_ID":"$AWS_SECRET_ACCESS_KEY" \
3	-H "x-amz-security-token: $AWS_SESSION_TOKEN" \
4	--aws-sigv4 "aws:amz:us-east-1:execute-api"

How to use curl for HTTP requests

Thomas Taylor — Mon, 15 May 2023 23:40:00 -0400

Curl is a command line utility that was created in 1998 for transferring data using URLs. Fundamentally, curl allows users to create network requests to a server by specifying a location in the form of a URL and adding optional data.

curl (short for “Client URL”) is powered by libcurl – a portable client-side URL transfer library written in C.

Why use the curl command?

Common use cases for curl include:

It can download and upload files
It has great error logging
It allows for quick endpoint testing
It provides granular details which aides debugging
It is portable and works with most operating systems
It is an actively managed open-source tool that is battled tested with a large user base

Base `curl` command usage

The basic syntax for a curl command is:

1curl [OPTIONS] [URL]

HTTP Requests using `curl`

Curl provides native support for all HTTP requests methods: GET, POST, PUT, PATCH, DELETE, etc.

For all the following examples, the JSONPlaceholder API will be used.

Make a `GET` request

1curl https://jsonplaceholder.typicode.com/posts/1

1curl -X GET https://jsonplaceholder.typicode.com/posts/1

NOTE: The -X flag is a shorthand for --request. It’s not required because the default value is GET.

Output:

1{
2  "userId": 1,
3  "id": 1,
4  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
5  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
6}

In addition, an optional Accept header may be used:

1curl \
2	-H "Accept: application/json" \
3	https://jsonplaceholder.typicode.com/posts/1

Make a `POST` request

1curl -X POST -d "title=foo&body=test" \
2	https://jsonplaceholder.typicode.com/posts

In the example above, a new post titled foo with a body of test is created.

The default Content-Type of the request is: application/x-www-form-urlencoded

If a JSON payload is required, then the Content-Type header can be explicitly included:

1curl -X POST \
2	-H 'Content-Type: application/json' \
3	-d '{"title":"foo","body":"bar"}' \
4	https://jsonplaceholder.typicode.com/posts

Alternatively, a file may be specified (post.json):

1{
2    "title": "foo",
3    "body": "bar"
4}

1curl -X POST \
2	-H 'Content-Type: application/json' \
3	-d @post.json \
4	https://jsonplaceholder.typicode.com/posts

Output:

1{
2  "title": "foo",
3  "body": "bar",
4  "id": 101
5}

Make a `PUT` request

Similarly to POST, a PUT can use all the same options:

1curl -X PUT -d "title=foo&body=test" \
2	https://jsonplaceholder.typicode.com/posts/1

In the example above, post 1 is re-titled to foo and given a new body of test.

The default Content-Type of the request is: application/x-www-form-urlencoded

If a JSON payload is required, then the Content-Type header can be explicitly included:

1curl -X PUT \
2	-H 'Content-Type: application/json' \
3	-d '{"title":"foo","body":"bar"}' \
4	https://jsonplaceholder.typicode.com/posts/1

Alternatively, a file may be specified (post.json):

1{
2    "title": "foo",
3    "body": "bar"
4}

1curl -X PUT \
2	-H 'Content-Type: application/json' \
3	-d @post.json \
4	https://jsonplaceholder.typicode.com/posts/1

Output:

1{
2  "title": "foo",
3  "body": "bar",
4  "id": 1
5}

Make a `PATCH` request

Similarly to POST and PUT, a PATCH can use all the same options:

1curl -X PATCH -d "title=foo2" \
2	https://jsonplaceholder.typicode.com/posts/1

In the example above, post 1 is re-titled to foo2.

The default Content-Type of the request is: application/x-www-form-urlencoded

If a JSON payload is required, then the Content-Type header can be explicitly included:

1curl -X PATCH \
2	-H 'Content-Type: application/json' \
3	-d '{"title":"foo2"}' \
4	https://jsonplaceholder.typicode.com/posts/1

Alternatively, a file may be specified (post.json):

1{
2    "title": "foo2"
3}

1curl -X PATCH \
2	-H 'Content-Type: application/json' \
3	-d @post.json \
4	https://jsonplaceholder.typicode.com/posts/1

Output:

1{
2  "userId": 1,
3  "id": 1,
4  "title": "foo2",
5  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
6}

Make a `DELETE` request

1curl -X DELETE https://jsonplaceholder.typicode.com/posts/1

Output:

1{}

Conclusion

curl is a powerful and versatile command line utility that offers a wide range of supporting features: multiple protocols, multiple options, data types, etc. HTTP requests are simple and reliable.

How to make a PATCH request using curl

Thomas Taylor — Sun, 14 May 2023 18:20:00 -0400

For more information about curl, checkout the “What is curl” article. This article will discuss how to interact with an API using PATCH requests through curl.

Make a simple `PATCH` request

The basic syntax for sending a PATCH request using curl is:

1curl -X PATCH https://example.com

The -X argument accepts an HTTP method for interacting with the server. For HTTP, valid values include: GET (default), POST, PUT, DELETE, etc.

NOTE: The -X flag is a shorthand for --request. Either can be used.

Make a `PATCH` request with data

Users can send data along with a PATCH request.

1curl -X PATCH -d "title=bar&body=foo" \
2	https://jsonplaceholder.typicode.com/posts/1

In the example above, a PATCH request is sent to the JSONPlaceholder API to update an existing post (1) with a title of bar and a body of foo.

The default Content-Type is application/x-www-form-urlencoded.

NOTE: The -d flag is a shorthand for --data. Either can be used.

Make a `PATCH` request with JSON body

Users may optionally send JSON data in their request payloads.

Making a `PATCH` request with JSON

1curl -X PATCH \
2	-H 'Content-Type: application/json' \
3	-d '{"title":"bar","body":"foo"}' \
4	https://jsonplaceholder.typicode.com/posts/1

In the example above, a PATCH request is sent to the JSONPlaceholder API to update an existing post (1) with a title of bar and a body of foo.

The -H flag accepts a Key: value string that represents a header. In the above case, it sets the content type: Content-Type: application/json.

NOTE: The -H flag is a shorthand for --header. Either can be used.

Making a `PATCH` request with a JSON file

In addition to text, the -d parameter accepts a file using the syntax @filename.

If a file is named post.json

1{ 
2	"title": "bar",
3	"body": "foo"
4}

then curl can pass on its contents:

1curl -X PATCH \
2	-H 'Content-Type: application/json' \
3	-d @post.json \
4	https://jsonplaceholder.typicode.com/posts/1

How to make a DELETE request using curl

Thomas Taylor — Sun, 14 May 2023 10:15:00 -0400

For more information about curl, checkout the “What is curl” article. This article will discuss how to interact with an API using DELETE requests through curl.

Make a simple `DELETE` request

The basic syntax for sending a DELETE request using curl is:

1curl -X DELETE https://example.com

NOTE: The -X flag is a shorthand for --request.

`DELETE` request example

1curl -X DELETE \
2    https://jsonplaceholder.typicode.com/posts/1

In the example above, a DELETE request is sent to the JSONPlaceholder API to delete a post with an id of 1.

How to make a PUT request using curl

Thomas Taylor — Sat, 13 May 2023 16:10:00 -0400

For more information about curl, checkout the “What is curl” article. This article will discuss how to interact with an API using PUT requests through curl.

Make a simple `PUT` request

The basic syntax for sending a PUT request using curl is:

1curl -X PUT https://example.com

The -X argument accepts an HTTP method for interacting with the server. For HTTP, valid values include: GET (default), POST, PUT, DELETE, etc.

NOTE: The -X flag is a shorthand for --request. Either can be used.

Make a `PUT` request with data

Users can send data along with a PUT request.

1curl -X PUT -d "title=newtitle&body=newbody" \
2	https://jsonplaceholder.typicode.com/posts/1

In the example above, a PUT request is sent to the JSONPlaceholder API to update an existing post (1) with a title of newtitle and a body of newbody.

The default Content-Type is application/x-www-form-urlencoded.

NOTE: The -d flag is a shorthand for --data. Either can be used.

Make a `PUT` request with JSON body

Users may optionally send JSON data in their request payloads.

Making a `PUT` request with JSON

1curl -X PUT \
2	-H 'Content-Type: application/json' \
3	-d '{"title":"newtitle","body":"newbar"}' \
4	https://jsonplaceholder.typicode.com/posts/1

In the example above, a PUT request is sent to the JSONPlaceholder API to update an existing post (1) with a title of newtitle and a body of newbody.

The -H flag accepts a Key: value string that represents a header. In the above case, it sets the content type: Content-Type: application/json.

NOTE: The -H flag is a shorthand for --header. Either can be used.

Making a `PUT` request with a JSON file

In addition to text, the -d parameter accepts a file using the syntax @filename.

If a file is named post.json

1{ 
2	"title": "newtitle",
3	"body": "newbody"
4}

then curl can pass on its contents:

1curl -X PUT \
2	-H 'Content-Type: application/json' \
3	-d @post.json \
4	https://jsonplaceholder.typicode.com/posts/1

How to make a GET request using curl

Thomas Taylor — Sat, 13 May 2023 11:50:00 -0400

For more information about curl, checkout the “What is curl” article. This article will discuss how to interact with an API using GET requests through curl.

Make a simple `GET` request

The basic syntax for sending a GET request using curl is:

1curl https://example.com

1curl -X GET https://example.com

NOTE: The -X flag is a shorthand for --request. It’s not required because the default value is GET.

Make a `GET` request for JSON data

If the server responds differently per the Accept header, users can specify for it to accept JSON data.

1curl \
2	-H "Accept: application/json" \
3	https://jsonplaceholder.typicode.com/posts/1

In the example above, a GET request is sent to the JSONPlaceholder API to fetch a post with an id of 1 with an Accept header of application/json.

The -H flag accepts a Key: value string that represents a header.

NOTE: The -H flag is a shorthand for --header. Either can be used.

This same technique may be used for other mime types:

application/xml
text/html
img/png
etc.

How to make a POST request using curl

Thomas Taylor — Sat, 13 May 2023 10:25:00 -0400

For more information about curl, checkout the “What is curl” article. This article will discuss how to interact with an API using POST requests through curl.

Make a simple `POST` request

The basic syntax for sending a POST request using curl is:

1curl -X POST https://example.com

The -X argument accepts an HTTP method for interacting with the server. For HTTP, valid values include: GET (default), POST, PUT, DELETE, etc.

NOTE: The -X flag is a shorthand for --request. Either can be used.

Make a `POST` request with data

Users can send data along with a POST request.

1curl -X POST -d "title=foo&body=test" \
2	https://jsonplaceholder.typicode.com/posts

In the example above, a POST request is sent to the JSONPlaceholder API to create a new post with a title of foo and a body of test.

The default Content-Type is application/x-www-form-urlencoded.

NOTE: The -d flag is a shorthand for --data. Either can be used.

Make a `POST` request with JSON body

Users may optionally send JSON data in their request payloads.

Making a `POST` request with JSON

1curl -X POST \
2	-H 'Content-Type: application/json' \
3	-d '{"title":"foo","body":"bar"}' \
4	https://jsonplaceholder.typicode.com/posts

In the example above, a POST request is sent to the JSONPlaceholder API to create a new post with a title of foo and a body of bar.

The -H flag accepts a Key: value string that represents a header. In the above case, it sets the content type: Content-Type: application/json.

NOTE: The -H flag is a shorthand for --header. Either can be used.

Making a `POST` request with a JSON file

In addition to text, the -d parameter accepts a file using the syntax @filename.

If a file is named post.json

1{ 
2	"title": "foo",
3	"body": "test"
4}

then curl can pass on its contents:

1curl -X POST \
2	-H 'Content-Type: application/json' \
3	-d @post.json \
4	https://jsonplaceholder.typicode.com/posts

What is the curl command

Thomas Taylor — Fri, 12 May 2023 00:55:00 -0400

curl (short for “Client URL”) is powered by libcurl – a portable client-side URL transfer library written in C.

Why use the curl command?

Common use cases for curl include:

It can download and upload files
It has great error logging
It allows for quick endpoint testing
It provides granular details which aides debugging
It is portable and works with most operating systems
It is an actively managed open-source tool that is battled tested with a large user base

Base `curl` command usage

The basic syntax for a curl command is:

1curl [OPTIONS] [URL]

Perform a `GET` request

The simplest usage is a default GET request for a website or API:

1curl https://pokeapi.co/api/v2/machine/1

Output:

1{"id":1,"item":{"name":"tm00","url":"https://pokeapi.co/api/v2/item/1288/"},"move":{"name":"mega-punch","url":"https://pokeapi.co/api/v2/move/5/"},"version_group":{"name":"sword-shield","url":"https://pokeapi.co/api/v2/version-group/20/"}}

Download a file

Downloading a file using the same name as the remote server:

1curl -O https://github.com/curl/curl/releases/download/curl-8_0_1/curl-8.0.1.tar.gz

Downloading a file and specifying a new name:

1curl -o curl.tar.gz https://github.com/curl/curl/releases/download/curl-8_0_1/curl-8.0.1.tar.gz

Conclusion

curl is a powerful and versatile command line utility that offers a wide range of supporting features: multiple protocols, multiple options, data types, etc.

Build a dead simple CLI in Bash

Thomas Taylor — Thu, 11 May 2023 01:45:00 -0400

Building a simple CLI in Bash may seem like a herculean task; however, getopts provides an easy-to-use interface out of the box! For this tutorial, we’ll be using the https://pokeapi.co/ to build a simple CLI for fetching resources from the Pokemon world.

What is `getopts`?

Given from the documentation,

The getopts utility shall retrieve options and option-arguments from a list of parameters. It shall support the Utility Syntax Guidelines 3 to 10, inclusive, described in XBD Utility Syntax Guidelines.

getopts is a command that makes defining options and option-arguments seamless using a list of parameters.

Use `getopts` to get `pokemon`

Let’s start by making the boilerplate:

 1#!/bin/bash
 2
 3usage() { 
 4  echo "Usage: $0 -p " 1>&2
 5  exit
 6}
 7
 8while getopts "hp:" o; do
 9  case "${o}" in
10    p) pokemon_name="$(echo "$OPTARG" | tr '[:upper:]' '[:lower:]')" ;;
11    h) usage ;;
12  esac
13done
14
15[[ -z "$pokemon_name" ]] && usage

As seen above, getopts accepts a parameter list – ie. hn:. Colons (:) are used to denote that an argument requires a value. In this case, h is as an option (-h) while n is as an option-argument (-n ). If $name is not provided, then the usage function is called.

Within the case statement, $OPTARG contains the option-argument value that is passed from the user. For good measure, it is piped through tr to lowercase it (click here for more information about casing strings in Bash).

Now, let’s add a simple curl command that calls the /pokemon api given the name:

 1#!/bin/bash
 2
 3usage() { 
 4  echo "Usage: $0 -p " 1>&2
 5  exit
 6}
 7
 8pokemon() {
 9  local name="$1"
10  curl -o "$name.json" https://pokeapi.co/api/v2/pokemon/$name \
11    -H "Accept: application/json"
12  exit
13}
14
15while getopts "hp:" o; do
16  case "${o}" in
17    p) pokemon_name="$(echo "$OPTARG" | tr '[:upper:]' '[:lower:]')" ;;
18    h) usage ;;
19  esac
20done
21
22[[ -z "$pokemon_name" ]] && usage
23[[ "$pokemon_name" ]] && pokemon "$pokemon_name"

The pokemon function is called with the $pokemon_name that was passed in.

1./poke.sh -p pikachu

The output is a json file that contains pikachu’s information.

Use `getopts` to get `items`

Building upon the last example, we can easily capture items:

 1#!/bin/bash
 2
 3usage() { 
 4  echo "Usage: $0 [-p ] [-i ]" 1>&2
 5  exit
 6}
 7
 8request() {
 9  local path="$1"
10  local file_name=$(echo $path | sed 's|.*/||')
11  curl -o "$file_name.json" https://pokeapi.co/api/v2/$path \
12    -H "Accept: application/json"
13}
14
15pokemon() {
16  request "pokemon/$1"
17  exit
18}
19
20item() {
21  request "item/$1"
22  exit
23}
24
25while getopts "hp:i:" o; do
26  case "${o}" in
27    p) pokemon_name="$(echo "$OPTARG" | tr '[:upper:]' '[:lower:]')" ;;
28    i) item_name="$(echo "$OPTARG" | tr '[:upper:]' '[:lower:]')" ;;
29    h) usage ;;
30  esac
31done
32
33[[ -z "$pokemon_name" && -z "$item_name" ]] && usage
34[[ "$pokemon_name" ]] && pokemon "$pokemon_name"
35[[ "$item_name" ]] && item "$item_name"

A few things happened:

I abstracted the curl to its own function named request
1. The file_name is parsed from the path. Ie. /items/master-ball -> master-ball.json
pokemon and item both call request
i: was added to the getopts string to allow the-i option-argument.

What’s next?

getopts is useful for building plain / simple CLIs. If something more advanced is necessary, bash is not the best language for the task.

Get file name or extension in Bash

Thomas Taylor — Wed, 10 May 2023 00:20:00 -0400

Retrieving the file name and extension in Bash can be completed using basename and shell parameter expansion.

Get the file name

Using the native unix basename command, the file name can be extracted from the path.

1file_path="/path/to/package.tar.gz"
2file_name=$(basename "$file_path")
3echo "$file_name"

Output:

1package.tar.gz

Get the file extension

Using shell parameter expansion, the prior example can include capturing the file extension:

1file_path="/path/to/package.tar.gz"
2file_name=$(basename "$file_path")
3file_ext="${file_name#*.}"
4echo "$file_ext"

Output:

1tar.gz

Match on file extension using a `case` statement

If there is conditional logic required for specific file extensions, a case statement may be used. In addition, this is standard sh (not exclusive to bash)

1file_path="/path/to/package.tar.gz"
2file_name=$(basename "$file_path")
3
4case "$file_name" in
5    *.tar.gz) echo "matched tar.gz!" ;;
6esac

Output:

1matched tar.gz!

Convert a string to lowercase in Bash

Thomas Taylor — Tue, 09 May 2023 00:10:00 -0400

Here are the top 3 ways for converting a string to lowercase in Bash.

Using `tr`

One of the most popular ways for lowercasing a string is using tr. This is a POSIX standard method for accomplishing the task.

1echo "I WANT this LoWeRcAsEd" | tr '[:upper:]' '[:lower:]'
2# Outputs: i want this lowercased

Using `awk`

Another POXIS standard is using awk’s native toLower method.

1echo "I WANT this LoWeRcAsEd" | awk '{print tolower($0)}'
2# Outputs: i want this lowercased

Using `Bash` 4.0

bash 4.0 natively supports converting strings to lowercase.

1toLower="I WANT this LoWeRcAsEd"
2echo "${toLower,,}"
3# Outputs: i want this lowercased

Use Linux terminal as a calculator

Thomas Taylor — Mon, 08 May 2023 01:20:00 -0400

Linux provides several methods for calculating expressions using the command line.

Using `bc` — basic calculator for command line

bc is a native Linux tool (originated on Unix systems in 1975) that stands for “basic calculator”. To use it, simply type bc followed by an expression. To quit, type quit.

 1> bc
 2Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
 3This is free software with ABSOLUTELY NO WARRANTY.
 4For details type `warranty'. 
 55 + 5
 610
 725 * 4
 8100
 9100 / 2 * 35
101750
112 * (2 * 5) + 15
1235
13quit

Additionally, variables, arrays, algebraic expressions, etc. can be used. Refer to the bc manual for more information.

Using `expr`

The expr command evaluates basic expressions.

1> expr 5 + 5
210
3> expr 25 \* 4
4100
5> expr 100 / 2 \* 35
61750

Keep in mind, it should be used for basic calculations. If something more advanced is required (like parenthesis), bc or arithmetic expansion may be more suitable.

Using arithmetic expansion

Arithmetic expansion evaluates an expression and substitutes the result. In combination with echo, it can be used as a calculator.

1> echo "$((5 + 5))"
210
3> echo "$((25 * 4))"
4100
5> echo "$((100 / 2 * 35))"
61750
7> echo "$((2 * (2 * 5) + 15))"
835

Using `qalc`

Qalc is a multi-purpose cross-platform desktop calculator. It provides versatility and supports many everyday needs: currency conversion, percent calculation, etc.

Install it on Ubuntu via apt:

1sudo apt install qalc

Example usage:

 1> sqrt(72)
 2
 3  sqrt(72) = 6 × √(2) ≈ 8.485281374
 4
 5> fibonacci(133) to hex
 6
 7  fibonacci(133) ≈ 0x90540BE2616C26F81F876B9
 8
 9> 5/3 + 3/7
10
11  (5 / 3) + (3 / 7) = 44/21 = 2 + 2/21 ≈ 2.095238095
12
13> 5 + 5
14
15  5 + 5 = 10
16
17> exit

Installing Docker on Ubuntu

Thomas Taylor — Sat, 06 May 2023 21:25:00 -0400

Installing Docker on Ubuntu-based Linux distributions: Linux Mint, Zorin, Pop OS, etc. is simple!

Installing Docker from CLI

Docker is available in the default repositories of apt. Install via:

1sudo apt install docker.io

Add permissions to the docker group by adding the current user:

1sudo usermod -a -G docker $(whoami)

Verify that everything worked:

1docker ps

Output:

1CONTAINER ID        IMAGE               COMMAND             CREATED

Troubleshooting

If the following error occurs after docker ps:

1Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json: dial unix /var/run/docker.sock: connect: permission denied

1. Check if Docker is running:

1sudo systemctl status docker

the status should show active (running).

If not, restart the docker service with the following command:

1sudo systemctl restart docker

2. Verify that the current user is apart of the docker group

1grep docker /etc/group

the output should show docker:x:135:USER with USER as the current user given from $(whoami). If not, add the current user to the docker group using the instructions above.

3. Verify that the tty session shows correct group membership

1id -nG

the output should contain docker in the list. If not, reboot.

4. Reboot

If all the prior steps were successful, simply reboot.

1sudo reboot

Test AWS credentials using AWS CLI

Thomas Taylor — Fri, 05 May 2023 23:55:00 -0400

Testing for AWS credentials is a straightforward operation.

Using `get-caller-identity`

There is a single API call that will always work regardless of permissions: GetCallerIdentity

1aws sts get-caller-identity

Output:

1{
2    "UserId": "AIDA...",
3    "Account": "123456789012",
4    "Arn": "arn:aws:iam::123456789012:user/thomas"
5}

The Arn value depends on the type of credentials, but mostly includes the human-friendly name.

In addition, checking the status is reliable: 0 for success, 255 for failure.

Split a list into evenly sized batches in Python

Thomas Taylor — Thu, 04 May 2023 21:30:00 -0400

Splitting a list into evenly sized chunks or batches can be accomplished using Python.

Using `itertools.batched`

In Python 3.12, itertools.batched is available for consumption.

1import itertools
2
3lst = ['mary', 'sam', 'joseph']
4print(list(itertools.batched(lst, 2))) # [('mary', 'sam'), ('joseph')]

Using `yield`

If the version of Python is lower than 3.12, the yield keyword may be used.

1def chunks(lst, n):
2    for i in range(0, len(lst), n):
3        yield lst[i:i + n]
4
5lst = ['mary', 'sam', 'joseph']
6print(list(chunks(lst, 2))) # [['mary', 'sam'], ['joseph']]

Using `itertools.islice`

The itertools.islice function can additionally be used.

1from itertools import islice
2
3def chunks(lst, n):
4    it = iter(lst)
5    return iter(lambda: tuple(islice(it, n)), ())
6
7lst = ['mary', 'sam', 'joseph']
8print(list(chunks(lst, 2))) # [('mary', 'sam'), ('joseph',)]

Delete local branches not on remote using Git

Thomas Taylor — Wed, 03 May 2023 23:55:00 -0400

After merging changes or working on temporary branches, users may retain a list of orphaned git branches on their local machine. Rather than deleting each branch one-by-one, running a simple script can clear them out.

Delete local branches not found on remote

The one-liner script to delete local branches not found one remote:

1git fetch -p && for branch in $(git for-each-ref --format '%(refname) %(upstream:track)' refs/heads | awk '$2 == "[gone]" {sub("refs/heads/", "", $1); print $1}'); do git branch -D $branch; done

This works by detected if a branch is [gone] from remote.

Access the current index in for loops in Python

Thomas Taylor — Tue, 02 May 2023 23:50:00 -0400

In Python, an iterable may be traversed using a simple for in loop:

1names = ["sally", "mary", "billy"]
2for n in names:
3	print(n)

However, what if the current index is desired?

Access the index in a Python `for` loop

The recommended pythonic solution is to use the built-in enumerate function.

1names = ["sally", "mary", "billy"]
2for i, n in enumerate(names):
3	print(i, n)

Output:

10 sally
21 mary
32 billy

For contrast, the unidiomatic way to access the current index is by using range:

1names = ["sally", "mary", "billy"]
2for i in range(len(names)):
3	print(i, names[i])

Output:

10 sally
21 mary
32 billy

Kill all processes that match a name

Thomas Taylor — Mon, 01 May 2023 23:40:00 -0400

Killing all processes that match a certain substring or name is easy using pkill.

Killing all processes using `pkill`

pkill is a native Linux tool that provides command line access for killing or reloading processes.

To kill all processes that match a pattern:

1pkill -f firefox

If pkill is not successful, trying adding the -9 flag.

1pkill -9 -f firefox

Confirmation prompt yes/no in Bash

Thomas Taylor — Sun, 30 Apr 2023 23:55:00 -0400

Asking for user confirmation in a bash script is easy!

Yes/No prompt with default no

If the default is No for an invalid character or space, simply check if the prompt answer is y. Using read, user input can be limited to a single character using the -n 1 parameter.

1read -r -p "Are you sure? [y/N]" -n 1
2echo # (optional) move to a new line
3if [[ "$REPLY" =~ ^[Yy]$ ]]; then
4    echo "Operation continues"
5fi

Similarly, a case statement may be used:

1read -r -p "Are you sure? [y/N]" -n 1
2echo # (optional) move to a new line
3case "$REPLY" in 
4  y|Y ) echo "Operation continues";;
5  * ) echo "Operation is cancelled";;
6esac

Create dictionaries or maps in Bash

Thomas Taylor — Sat, 29 Apr 2023 23:55:00 -0400

bash 4.0 natively supports creating hash tables / maps / dictionaries.

Using maps in Bash

Variables in bash are loosely typed; however, declaring provides a type-like behavior. One of the supported types in bash is an associative array (-A).

To declare an associative array:

1declare -A scores

To declare and instantiate in a single line:

1declare -A scores=( ["peter"]=95 ["sally"]=98 )

Add key/value pairs

Adding key/value pairs to an existing associate array:

1scores["tom"]=85
2
3# using variables
4name="brandon"
5score=50
6scores[$name]=$score

Retrieve key/value pairs

Retrieving key/value pairs from an existing associative array:

1name="peter"
2echo ${scores[$name]} # 95

Update an existing key/value pair

Updating an existing key/value pair in an existing associative array:

1name="peter"
2scores[$name]=100
3echo ${scores[$name]} # 100

Check if a key exists

1name="peter"
2if [ -v scores[$name] ]; then
3    echo "$name exists in the map"
4fi
5# Output: peter exists in the map

Remove a key

1name="peter"
2unset scores[$name]

Iterate map

1for key in "${!scores[@]}"; do
2    echo "$key ${scores[$key]}"
3done

Output:

1tom 85
2brandon 50
3sally 98
4peter 100

Get seconds since Epoch in Bash

Thomas Taylor — Fri, 28 Apr 2023 23:50:00 -0400

bash 5.0 natively supports retrieving seconds since Epoch using a variable.

Native `bash` method

1echo $EPOCHSECONDS # 1682740449

If seconds are preferred:

1echo $EPOCHREALTIME # 1682740449.252199

If milliseconds are preferred:

1echo $(( ${EPOCHREALTIME/./} / 1000 )) # 1682740449252

Using `date`

In addition to using the native bash method, date may be used.

According to the date manual:

%s seconds since the Epoch (1970-01-01 00:00 UTC)

1echo $(date +%s) # 1682740326

If milliseconds seconds are preferred:

1echo $(date +%s%3N) # 1682740449252

Catch and handle errors in Bash

Thomas Taylor — Thu, 27 Apr 2023 22:10:00 -0400

Bash does not natively support a try/catch syntax for errors; however, there are options for handling errors within a script.

Try and Catch errors in Bash

For the first technique, simply detect if there’s a non-zero status by using the following syntax:

1if ! command_call; then
2    echo "command_call did not complete successfully"
3fi

In addition, an || (or) expression may be used instead:

1command_call || echo "command_call did not complete successfully"

For explicitness, the full syntax for a try/catch in Bash is:

1if command_call ; then # try
2    echo "tried and was successful"
3else # catch
4    echo "command_call did not complete successfully"
5fi

If command_call is not successful, the echo statement will be invoked in all scenarios.

Error handling in Bash

For the second technique, bash provides a native variable for reporting exit codes: $?. Some users may want to catch and handle specific exit codes.

1command_call
2
3status=$?
4if [ $status -eq 1 ]; then
5    echo "General exception"
6elif [ $status -eq 127 ]; then
7    echo "The command could not be found!"
8fi

Output: ids.txt:

11
22

Using the redirection operator (>), the jq results are written to a file.

Python title method in JavaScript

Thomas Taylor — Sun, 23 Apr 2023 23:25:00 -0400

What is the equivalent for the Python string title method in JavaScript? Python’s native str.title() method returns a string with title-casing— i.e. the first letter of each word is capitalized.

`str.title()` method in JavaScript

Unfortunately, JavaScript does not contain a standardized str.title() method; however, a function can be created that implements equivalent behavior.

In Python, here are a few examples of the str.title() method:

1print("WeIrD cAsInG".title()) # Weird Casing
2print("HELLO WORLD".title()) # Hello World
3print("nospaceHERE".title()) # Nospacehere
4print("there's a snake in my boot".title()) # There'S A Snake In My Boot

This is an equivalent JavaScript implementation:

1function title(str) {
2  return str.toLowerCase().split(' ').map(w => w.charAt(0).toUpperCase() + w.slice(1)).join(' ');
3}
4
5console.log(title("WeIrD cAsInG")); // Weird Casing
6console.log(title("HELLO WORLD")); // Hello World
7console.log(title("nospaceHERE")); // Nospacehere
8console.log(title("there's a snake in my boot")); // There'S A Snake In My Boot

Deep clone an object in JavaScript

Thomas Taylor — Sat, 22 Apr 2023 22:15:00 -0400

Deep cloning an object in JavaScript was natively added in Node 17.

Deep clone an object using `structuredClone`

Deepling cloning an object is completed using structuredClone.

 1const a = {
 2  col1: 'val1',
 3  col2: {
 4      nestedObjected: {
 5          col1: 'val1'
 6      }
 7  }
 8}
 9
10const b = structuredClone(a);
11console.log(b); 
12// {"col1":"val1","col2":{"nestedObjected":{"col1":"val1"}}}

Deep clone an object using `JSON`

If the structuredClone method is not available and the user does not want to add an external library, using JSON may satisfy the requirement. If the following types are not used:

Date
undefined
Infinity
Regular expressions
Maps
Sets
Blobs
other compex types within an object

a simple one liner may be sufficient.

 1const a = {
 2  col1: 'val1',
 3  col2: {
 4      nestedObjected: {
 5          col1: 'val1'
 6      }
 7  }
 8}
 9
10const b = JSON.parse(JSON.stringify(a))
11console.log(b); 
12// {"col1":"val1","col2":{"nestedObjected":{"col1":"val1"}}}

Search across all object values in an array in JavaScript

Thomas Taylor — Sat, 22 Apr 2023 11:45:00 -0400

If there is a need to search across all object values in an array in JavaScript, a combination of filter, entries, and some can be used.

How to search across object values

In a frontend data table, rows may contain information about people:

 1const people = [
 2  {
 3    "id": 5,
 4    "firstName": "Buzz",
 5    "lastName": "Lightyear2"
 6  },
 7  {
 8    "id": 2,
 9    "firstName": "Buzz",
10    "lastName": "Lightyear"
11  },
12  {
13    "id": 1,
14    "firstName": "Woody",
15    "lastName": "Pride"
16  },
17  {
18    "id": 3,
19    "firstName": "Potato",
20    "lastName": "Head"
21  },
22  {
23    "id": 4,
24    "firstName": "Slinky",
25    "lastName": "Dog"
26  }
27];

To provide a “search” across all of the values, use the following:

1const query = "Slinky".toLowerCase();
2const results = people.filter(o => Object.entries(o).some(e => String(e[1]).toLowerCase().includes(query)));
3console.log(results); // [ { id: 4, firstName: 'Slinky', lastName: 'Dog' } ]

If id or other fields are not preferable in the search, use the rest operator.

1const query = "Slinky".toLowerCase();
2const results = people.filter(o => {
3  const { id, ...rest } = o;
4  return Object.entries(rest).some(e => String(e[1]).toLowerCase().includes(query))
5});
6console.log(results); // [ { id: 4, firstName: 'Slinky', lastName: 'Dog' } ]

Sort array of objects by single or multiple keys in JavaScript

Thomas Taylor — Thu, 20 Apr 2023 23:45:00 -0400

In TypeScript or JavaScript, sorting an array of objects is made easy in ES6/ES2015.

Given an array of objects named people, sort them based on different criteria.

 1const people = [
 2  {
 3    "id": 5,
 4    "firstName": "Buzz",
 5    "lastName": "Lightyear2"
 6  },
 7  {
 8    "id": 2,
 9    "firstName": "Buzz",
10    "lastName": "Lightyear"
11  },
12  {
13    "id": 1,
14    "firstName": "Woody",
15    "lastName": "Pride"
16  },
17  {
18    "id": 3,
19    "firstName": "Potato",
20    "lastName": "Head"
21  },
22  {
23    "id": 4,
24    "firstName": "Slinky",
25    "lastName": "Dog"
26  }
27];

Sorting an array of objects by a single key

Sort ascending by number

1// sort ascending by id
2people.sort((a, b) => a.id - b.id)

Sort ascending by string

1// sort ascending by firstName
2people.sort((a, b) => a.firstName.localeCompare(b.firstName))

Sort descending by number

1// sort descending by id
2people.sort((a, b) => b.id - a.id)

Sort descending by string

1// sort descending by firstName
2people.sort((a, b) => b.firstName.localeCompare(a.firstName))

Sorting array of objects by multiple keys

If a secondary key is needed for additional sorting, the same sort method can be used.

1people.sort(
2    (a, b) => a.firstName.localeCompare(b.firstName) ||
3    a.lastName.localeCompare(b.lastName)
4);

the or logical operator || chains multiple sort options.

Function and method overloading in TypeScript

Thomas Taylor — Wed, 19 Apr 2023 23:55:00 -0400

In Typescript, the declaration of an overload is defined by writing overload signatures.

To do this, write some number of function signatures (usually two or more), followed by the body of the function

Be aware that the implementation signature type should be generic enough to include the overload signatures.

For example, an improper implementation signature:

1function fn(x: string): string;
2// Return type isn't right
3function fn(x: number): boolean;
4// This overload signature is not compatible with its implementation signature.
5function fn(x: string | number) {
6  return  "oops";
7}

Function overloading

Define a speak function that accepts a string or a number in the signature.

 1function speak(message: string): string;
 2function speak(num: number): string;
 3function speak(m: unknown) {
 4  if (typeof m === 'string') {
 5    return m;
 6  } else  if (typeof m === 'number') {
 7    return  `${m}`
 8  }
 9  throw  new  Error('Unable to greet');
10}
11console.log(speak(15)); // "15"
12console.log(speak("a message")); // "a message"

In the implementation signature, m is of value unknown and covers both overloaded signatures.

Method overloading

Using the same technique as before, the speak method can be included in a class.

 1class  Speaker {
 2  speak(message: string): string;
 3  speak(num: number): string;
 4  speak(m: unknown): unknown {
 5    if (typeof m === 'string') {
 6      return m;
 7    } else  if (typeof m === 'number') {
 8      return  `${m}`
 9    }
10    throw  new  Error('Unable to greet');
11  }
12}
13console.log(new  Speaker().speak(15));
14console.log(new  Speaker().speak("a message"));

Dynamic variable names in Bash

Thomas Taylor — Tue, 18 Apr 2023 23:50:00 -0400

In Bash, dynamic variables can be created using indirect expansion.

How to create a dynamic variable name

In a CI container, there may be a need to delineate between different environment variable values using a prefix for a variable name.

1#!/bin/sh
2# In the container at runtime:
3# export PROD_TOKEN="123"
4env="PROD"
5secret="${env}_TOKEN"
6echo ${!secret}

Output:

In the example above, the secret value is PROD_TOKEN after expansion. Using indirect expansion, ${!secret} references 123.

Parallel commands in Bash

Thomas Taylor — Mon, 17 Apr 2023 23:45:00 -0400

Bash natively supports a wait command.

Run parallel processes

Using default `wait` behavior

By default, wait will wait for all child processes to complete before continuing an execution. For most cases, this is sufficient.

1sleep 2 && echo "hi" &
2sleep 2 && echo "hi2" &
3
4# Wait for child processes to finish
5wait

Using the & character at the ending of a line sends the process to the background to complete. Without wait, the the script would end.

The script will complete in 2 seconds.

Output variation 1:

1hi
2hi2

Output variation 2:

1hi2
2hi

Using `wait` with process ids

wait allows for a process id argument value. This may be useful for establishing chained waits.

In the example below, the “hi3” task depends on “hi”; however “hi2” may run in parallel with “hi”.

 1sleep 2 && echo "hi" &
 2hi_pid=$!
 3
 4# "hi2" does not require "hi"
 5sleep 2 && echo "hi2" &
 6
 7wait $hi_pid
 8
 9# "hi3" requires "hi"
10sleep 2 && echo "hi3" &
11
12wait

Output:

1hi2
2hi
3hi3

The process id of the last execution is returned using $!.

Return a string from a function in Bash

Thomas Taylor — Sun, 16 Apr 2023 23:45:00 -0400

Bash allows returning status codes (integers) from functions; however, how can a string be returned?

Returning a string from a function

In Bash, command substitution can be used to capture the stdout of a function.

1f(){
2  local var="test"
3  echo $var
4}
5
6result=$(f)
7echo "f returned $result"

Output

1f returned test

Returning a string and status from a function

If the standard output and status code need capturing, the example above can be modified to include a return status.

1f(){
2  local var="test"
3  echo $var
4  return 10
5}
6
7result=$(f)
8status=$?
9echo "f returned $result with status code $status"

Output

1f returned test with status code 10

jq starts with query

Thomas Taylor — Sat, 15 Apr 2023 23:58:00 -0400

If a user wants to search a json payload using a starts_with or “begins with” or “has prefix” query, jq natively supports it.

How to query using `startswith`

Given a json array, searching for a specific string that startswith a prefix is easy.

 1{
 2  "people": [
 3    {
 4      "firstName": "thomas",
 5      "age": 25
 6    },
 7    {
 8      "firstName": "sally",
 9      "age": 28
10    }
11  ]
12}

Find each person if their name begins with sal.

1jq -r '.people[]|select(.firstName | startswith("sal"))' people.json

Output

1{
2  "firstName": "sally",
3  "age": 28
4}

Getters and setters in Python

Thomas Taylor — Fri, 14 Apr 2023 23:45:00 -0400

Writing getter and setter methods in Python is seamless using the @property decorator.

Using the `@property` decorator for getter and setter methods

Directly from the documentation:

 1class C(object):
 2    def __init__(self):
 3        self._x = None
 4
 5    @property
 6    def x(self):
 7        """I'm the 'x' property."""
 8        print("getter of x called")
 9        return self._x
10
11    @x.setter
12    def x(self, value):
13        print("setter of x called")
14        self._x = value
15
16    @x.deleter
17    def x(self):
18        print("deleter of x called")
19        del self._x
20
21
22c = C()
23c.x = 'foo'  # setter called
24foo = c.x    # getter called
25del c.x      # deleter called

An implementation of this method could be a product where a user can set a price after its instantiation.

 1class Product:
 2    def __init__(self, name):
 3        self._name = name
 4
 5    @property
 6    def price(self):
 7        return self._price
 8
 9    @price.setter
10    def price(self, new_price):
11        if new_price < 0 or not isinstance(new_price, float):
12            raise Exception("price must be greater than 0 and a float")
13        self._price = new_price
14
15    @price.deleter
16    def price(self):
17        del self._price
18
19
20p1 = Product("one")
21p1.price = 5.5

Difference between is and equal in Python

Thomas Taylor — Thu, 13 Apr 2023 23:30:00 -0400

In Python, there are two operators for determining equality: is and ==; however, what are the differences between them and when should one be used over the other?

What is the difference between the `is` and `==`

The is operator checks for object identity
The == operator checks for equality

Here is an example demonstrating the differences:

1foo = [1, 2, 3]
2bar = foo
3print(foo is bar) # True
4print(foo == bar) # True

In the example above, bar points to the same object reference as foo. Because foo and bar point to the same object, is reports true.

If a copy of foo’s list is assigned to bar,

1bar = foo[:]
2print(foo is bar) # False
3print(foo == bar) # True

is reports False since the two variables do not point to the same object.

When to use which?

As a general rule of thumb, use the is operator for the following use-cases:

Verify if two objects are the same object (not just the value)
Comparison against constants: None.

As stated in the PEP 8 style guide,

Comparisons to singletons like None should always be done with is or is not, never the equality operators.

Outside of those two use cases, default to using the == or != operators.

Promise all in Python

Thomas Taylor — Wed, 12 Apr 2023 23:55:00 -0400

In JavaScript, promise.all() returns a single promise given an iterable of promises. The returned promise is fulfilled when all the iterable promises are fulfilled. If one promise is rejected, the resulting promise is also rejected.

1try {
2  const values = await Promise.all([promise1, promise2]);
3  console.log(values); // [resolvedValue1, resolvedValue2]
4} catch (err) {
5  console.log(error); // rejectReason of any first rejected promise
6}

How to use `promise.all()` in Python

Python natively supports similar functionality to promise.all(). The method is named asyncio.gather().

 1import asyncio
 2
 3async def some_function():
 4	await asyncio.sleep(1)
 5	return 1
 6
 7async def main():
 8    values = await asyncio.gather(
 9        some_function(),
10        some_function()
11    )
12    print(values) # [1, 1]
13
14asyncio.run(main())

Enums in Python

Thomas Taylor — Tue, 11 Apr 2023 23:45:00 -0400

In Python 3.4, Enum support was added.

How to implement an `Enum`

Implementing an Enum in Python is trivial.

 1from enum import Enum
 2
 3
 4class PizzaSize(Enum):
 5    SMALL = 0
 6    MEDIUM = 1
 7    LARGE = 2
 8
 9
10print(PizzaSize.SMALL) # PizzaSize.SMALL
11print(PizzaSize.SMALL.value) # 0
12print(PizzaSize.SMALL.name) # SMALL

As a shortcut, a range value may be applied:

 1from enum import Enum
 2
 3
 4class PizzaSize(Enum):
 5    SMALL, MEDIUM, LARGE = range(3)
 6
 7
 8print(PizzaSize.SMALL)  # PizzaSize.SMALL
 9print(PizzaSize.SMALL.value)  # 0
10print(PizzaSize.SMALL.name)  # SMALL

Enums in Golang

Thomas Taylor — Mon, 10 Apr 2023 23:50:00 -0400

GO does not natively support an enum keyword; however, an enum-like syntax can be implemented using the help of iota.

What is `iota`?

iota is an identifier that auto increments constants.

In the following code, the constant number is manually incremented:

1const (
2    ZERO = 0
3    ONE = 1
4    TWO = 2
5)

however, it can be simplified to:

1const (
2	ZERO = iota // 0
3	ONE         // 1
4	TWO         // 2
5)

If desired, the iota keyword can be used on each line:

1const (
2	ZERO = iota // 0
3	ONE  = iota // 1
4	TWO  = iota // 2
5)

If a const keyword is used again, the iota will reset to 0.

 1const (
 2	ZERO = iota // 0
 3	ONE         // 1
 4	TWO         // 2
 5)
 6
 7const (
 8	ZEROAGAIN = iota // 0
 9	ONEAGAIN         // 1
10	TWOAGAIN         // 2
11)

Implementing an enum in GO

Using a combination of iota, int and const, the concept of an enum can be applied.

 1package main
 2
 3import "fmt"
 4
 5type PizzaSize int
 6
 7const (
 8	SMALL  PizzaSize = iota + 1 // 1
 9	MEDIUM                      // 2
10	LARGE                       // 3
11)
12
13func (s PizzaSize) String() string {
14	return [...]string{"Small", "Medium", "Large"}[s-1]
15}
16
17func (s PizzaSize) Index() int {
18	return int(s)
19}
20
21func main() {
22	size := SMALL
23	fmt.Println(size)          // Print : Small
24	fmt.Println(size.String()) // Print : Small
25	fmt.Println(size.Index())  // Print : 1
26}

AWS CLI starts with query

Thomas Taylor — Sun, 09 Apr 2023 19:30:00 -0400

If a user wants to search AWS resources using a starts_with or “begins with” or “has prefix” query, the AWS CLI natively supports it.

How to query using `starts_with`

Using AWS CloudFormation as an example, the AWS CLI allows for the following query syntax:

1export prefix=""
2aws cloudformation describe-stacks \
3	--query "Stacks[?starts_with(StackName, `$prefix`) == `true`].StackName"

This outputs all of the CloudFormation stack names that start with a particular prefix.

`starts_with`

The starts_with syntax is included in the JMESPath specification.

1boolean starts_with(string $subject, string $prefix)

Returns true if the $subject starts with the $prefix, otherwise this function returns false.

Executing Python scripts with a shebang

Thomas Taylor — Sat, 08 Apr 2023 10:50:00 -0400

The shebang line informs the operating system’s shell where to find the interpreter. The program loader will execute the specified interpreter with the same arguments that were initially passed.

Add a shebang (`#!`) to a Python script

How to add a shebang (#!) to a Python 3 file:

1#!/usr/bin/env python3
2
3print("hello world")

In addition to the normal execution by calling the interpreter directly:

1python3 ./hello.py

the file can be called like this:

1./hello.py

Note: The executable may not have executable permissions by the user. To enable for this, simply run:

1chmod +x ./hello.py

gyp no xcode or clt version detected

Thomas Taylor — Fri, 07 Apr 2023 22:25:00 -0400

On a fresh update of MacOS, an error may occur:

1gyp: No Xcode or CLT version detected!  
2gyp ERR! configure error

Solution 1: Reset xcode

Try simply running the following command:

1sudo xcode-select --reset

Solution 2: Reinstall xcode

If the prior solution does not work, a complete reinstall of xcode may be necessary.

Find the xcode path:

1xcode-select --print-path

Delete the xcode directory

1sudo rm -rf

Install xcode

1xcode-select --install

Check for valid json string or file with jq

Thomas Taylor — Thu, 06 Apr 2023 21:30:00 -0400

Verifying if a string or file is valid json is easy with jq.

Check if a string is valid `json`

Using the -e argument, jq will return a 0 if the last output was neither false nor null.

1if jq -e . >/dev/null 2>&1 <<<'{"test": "value"}'; then
2    echo "parsed successfully"
3else
4    echo "could not parse json string"
5fi

In addition, negating the above condition may increase readability:

1if ! jq -e . >/dev/null 2>&1 <<<'{"test": "value"}'; then
2    echo "could not parse json string"
3    exit 1
4fi

Using a variable:

1if ! jq -e . >/dev/null 2>&1 <<<"$json"; then
2    echo "could not parse json string"
3    exit 1
4fi

Check if a file is valid `json`

Using the same technique from above:

1if ! jq -e . >/dev/null 2>&1 <<<$(cat invalid.json); then
2    echo "could not parse json file"
3    exit 1
4fi

the invalid.json file:

1{ "invalid"

will output:

1could not parse json file

Patch operations for updating api resources with the AWS CLI

Thomas Taylor — Wed, 05 Apr 2023 23:45:00 -0400

When updating API Gateway resources using the AWS CLI, some commands include a --patch-operations argument.

For example, the update-base-path-mapping allows an optional --patch-operations. For more information about which resources are supported via patch operations, the AWS documentation is here.

How to use the `--patch-operations` argument

In the case for updating a base path mapping, a user may need to supply a restapiId and stage value.

The JSON array:

 1[
 2  {
 3    "op": "replace",
 4    "path": "/restapiId",
 5    "value": "id"
 6  },
 7  {
 8    "op": "replace",
 9    "path": "/stage",
10    "value": "prod"
11  }
12]

The aws apigateway update-base-path-mapping allows for a stringified json array value for the --patch-operations argument:

1domain_name="example.com"
2base_path="some-path"
3api_id="id"
4stage="prod"
5aws apigateway update-base-path-mapping \
6	--domain-name $domain_name \
7	--base-path $base_path \
8	--patch-operations "[{\"op\":\"replace\",\"path\":\"/restapiId\",\"value\":\"$api_id\"},{\"op\":\"replace\",\"path\":\"/stage\",\"value\":\"$stage\"}]"

The same --patch-operations argument format applies to the other API Gateway commands as well.

AWS Lambda caching in Python

Thomas Taylor — Tue, 04 Apr 2023 23:30:00 -0400

Caching between lambda invocations may be useful for certain scenarios:

Authentication
Rate limits with external services
Speed

How to cache between lambda invocations

In Python, it’s as simple as modifying a global variable.

1obj = None
2
3def lambda_handler(event):
4    global obj
5    if not obj:
6        obj = YourClass(event)

Upon initial run of a lambda function, obj will be set to the output of YourClass(event). On subsequent warm starts, the lambda function will reuse the global obj from the previous lambda run. More information regarding warm vs cold starts can be found here.

Calculate the middle between two dates in JavaScript

Thomas Taylor — Mon, 03 Apr 2023 23:35:00 -0400

In TypeScript or JavaScript, determining the midpoint between two dates is easy using the native Date library.

Determine the middle between two dates

1const startDate = new Date('Mar 10 2023');
2const endDate = new Date('Mar 15 2023');
3
4const midDate = new Date((startDate.getTime() + endDate.getTime()) / 2);
5console.log(midDate) // 2023-03-12T12:00:00.000Z

In the example above, getTime() returns the number of seconds since epoch. Adding the seconds and dividing by two gives the midpoint.

Python dataclass validation

Thomas Taylor — Sun, 02 Apr 2023 23:45:00 -0400

Creating dataclasses in Python is simplistic; however, what if additional validation needs to be completed during initialization?

Creating a dataclass

The following example has a class named Person that models information about an individual:

1from dataclasses import dataclass
2
3@dataclass
4class Person:
5    first_name: str
6    last_name: str
7    age: int

This class can be instantiated in several ways:

1john = Person(first_name="john", last_name="doe", age=35)
2sally = Person(first_name="sally", last_name="may", age=-20)
3zachary = Person(first_name="zachary", last_name="taylor", age=12.5)
4print(john) # Person(first_name='john', last_name='doe', age=35)
5print(sally) # Person(first_name='sally', last_name='may', age=-20)
6print(zachary) # Person(first_name='zachary', last_name='taylor', age=12.5)

How to validate inputs for dataclasses

In the previous example, sally’s age was a non-positive integer and zachary’s age was a float value. To resolve erroneous input, a __post_init__ method can be used:

 1from dataclasses import dataclass
 2
 3@dataclass
 4class Person:
 5    first_name: str
 6    last_name: str
 7    age: int
 8
 9    def __post_init__(self):
10        if not isinstance(self.age, int):
11            raise ValueError("age is not an int")
12        if self.age <= 0:
13            raise ValueError("age must be a positive integer greater than 0")

sally’s error:

1ValueError: age must be a positive integer greater than 0

zachary’s error:

1ValueError: age is not an int

Check if a variable is an array in JavaScript

Thomas Taylor — Sat, 01 Apr 2023 23:20:00 -0400

In TypeScript or JavaScript, checking if a variable is an array can be completed in two ways.

Using the `Array.isArray()`

Using the native Array.isArray() method, a boolean will be returned.

1const names = ["sally", "billy", "mary"];
2const notAnArray = 'str';
3console.log(Array.isArray(names)); // true
4console.log(Array.isArray(notAnArray)); // false

Using the `instanceOf` operator

As an alternative option, instanceOf will yield the same results.

1const names = ["sally", "billy", "mary"];
2const notAnArray = 'str';
3console.log(names instanceof Array); // true
4console.log(notAnArray instanceof Array); // false

Pull latest changes from submodules in Git

Thomas Taylor — Fri, 31 Mar 2023 21:50:00 -0400

Pulling the latest changes for all submodules in a repository is made easy through Git.

Pull all submodules after checkout

After an initial checkout, the the submodules can be updated using the following command:

1git submodule update --init --recursive

Pull all submodules changes

In Git versions 1.8.2 and higher, a --remote option was added to pull the latest changes from remote branches.

1git submodule update --recursive --remote

In addition, the following commands will work as well:

1git submodule update --recursive

1git pull --recurse-submodules

Reset a single file in Git

Thomas Taylor — Thu, 30 Mar 2023 20:45:00 -0400

Resetting a single file is possible using Git.

How to restore a single file

In Git versions 2.23 and higher, git restore natively supports resetting a single file.

1git restore path/to/file.txt

The above restores path/to/file.txt to the prior commit.

In addition, restoring from another branch is possible:

1git restore -s my-feature-branch pathTo/MyFile

The -s option specifies a source (my-feature-branch in this case).

How to reset a single file

In Git versions that predate 2.23, the following command will reset a single file:

1git checkout HEAD -- path/to/file.txt

Anything after the -- is treated as a filename in unix.

Match everything between two characters in Regex

Thomas Taylor — Wed, 29 Mar 2023 12:50:00 -0400

Matching everything between two delimiters in Regex is supported. In the examples below, the [ and ] characters were used as example delimiters.

Match between characters using lookahead / lookbehind

Using lookbehind and lookahead, the following patterns will match between two delimiters:

1(?<=\[)(.*?)(?=\])

Using regex101, the explanation can summarized as:

Match the non-greedy capture group (.*?) that is preceded by a [ and is followed by ] but do not capture the delimiters.

Match between characters without lookahead / lookbehind

Without using lookbehind or lookahead, the following pattern will match contents between two delimiters including the delimiters:

1\[(.*?)\]

Here is the regex101 link.

To remove the delimiters, the first captured group of each match will need to be used.

JSON prettify in Python

Thomas Taylor — Tue, 28 Mar 2023 00:20:00 -0400

Using the native json Python module, JSON data can be pretty-printed.

Pretty print JSON string in Python

1import json
2data = '{"users": [{"id":1,"name":"Thomas"},{"id":2,"name":"Sally"}]}'
3obj = json.loads(data)
4print(json.dumps(obj, indent=2))

Output:

 1{
 2  "users": [
 3    {
 4      "id": 1,
 5      "name": "Thomas"
 6    },
 7    {
 8      "id": 2,
 9      "name": "Sally"
10    }
11  ]
12}

The json.loads() method creates a Python dictionary from a given json string.
The json.dumps() method outputs a json string from a given Python dictionary. In addition, the indent parameter defines the level of the indent in the resulting string.

Pretty print JSON file data in Python

1import json
2
3with open("users.json", "r") as json_file:
4    obj = json.load(json_file)
5
6print(json.dumps(obj, indent=2))

Output:

 1{
 2  "users": [
 3    {
 4      "id": 1,
 5      "name": "Thomas"
 6    },
 7    {
 8      "id": 2,
 9      "name": "Sally"
10    }
11  ]
12}

The json.load() method creates a Python dictionary from a given file.
The json.dumps() method outputs a json string from a given Python dictionary. In addition, the indent parameter defines the level of the indent in the resulting string.

Specify a log group for a lambda function in AWS CloudFormation

Thomas Taylor — Mon, 27 Mar 2023 01:50:00 -0400

By default, lambda functions will create their own log groups if they are given proper permissions.

Specifying a log group for a Lambda Function

The following CloudFormation template creates two resources:

AWS::IAM::Role
AWS::Lambda::Function

 1AWSTemplateFormatVersion: '2010-09-09'
 2Resources:
 3  LambdaExecutionRole:
 4    Type: 'AWS::IAM::Role'
 5    Properties:
 6      AssumeRolePolicyDocument:
 7        Version: '2012-10-17'
 8        Statement:
 9        - Effect: Allow
10          Principal:
11            Service:
12            - lambda.amazonaws.com
13          Action:
14          - sts:AssumeRole
15      Policies:
16      - PolicyName: logs
17        PolicyDocument:
18          Version: '2012-10-17'
19          Statement:
20          - Effect: Allow
21            Action:
22            - logs:CreateLogGroup
23            - logs:CreateLogStream
24            - logs:PutLogEvents
25            Resource: '*'
26  LambdaFunction:
27    Type: 'AWS::Lambda::Function'
28    Properties:
29      FunctionName: 'LambdaTest'
30      Handler: index.handler
31      Runtime: nodejs18.x
32      Role: !GetAtt LambdaExecutionRole.Arn
33      Code:
34        ZipFile: |
35          exports.handler = async (event) => {
36            return 'Hello World!'
37          }

The LambdaTest function defaults to writing log events to a group named /aws/lambda/LambdaTest with a default retention of Never expire. To circumvent this behavior, a AWS::Logs::LogGroup resource can be explicitly created.

 1AWSTemplateFormatVersion: '2010-09-09'
 2Resources:
 3  LambdaFunctionLogGroup:
 4      Type: 'AWS::Logs::LogGroup'
 5      Properties:
 6        LogGroupName: "/aws/lambda/LambdaTest"
 7        RetentionInDays: 1
 8  LambdaFunction:
 9    Type: 'AWS::Lambda::Function'
10    DependsOn: LambdaFunctionLogGroup
11    Properties:
12      FunctionName: 'LambdaTest'
13      Handler: index.handler
14      Runtime: nodejs18.x
15      Role: !GetAtt LambdaExecutionRole.Arn
16      Code:
17        ZipFile: |
18          exports.handler = async (event) => {
19            return 'Hello World!'
20          }          
21  LambdaExecutionRole:
22    Type: 'AWS::IAM::Role'
23    Properties:
24      AssumeRolePolicyDocument:
25        Version: '2012-10-17'
26        Statement:
27        - Effect: Allow
28          Principal:
29            Service:
30            - lambda.amazonaws.com
31          Action:
32          - sts:AssumeRole
33      Policies:
34      - PolicyName: logs
35        PolicyDocument:
36          Version: '2012-10-17'
37          Statement:
38          - Effect: Allow
39            Action:
40            - logs:CreateLogGroup
41            - logs:CreateLogStream
42            - logs:PutLogEvents
43            Resource: !GetAtt LambdaFunctionLogGroup.Arn

A few aspects are different:

A new AWS::Logs::LogGroup was created with a retention period of 1 day
The LambdaTest resource uses DependsOn: LambdaFunctionLogGroup
The LambdaExecutionRole only allows writing logs to the LambdaFunctionLogGroup.Arn resource

Warning: If the log group already exists from a prior deployment, it will need to be deleted.

What does PHONY mean in a Makefile

Thomas Taylor — Sun, 26 Mar 2023 01:35:00 -0400

What does .PHONY mean in a Makefile? The make manual describes the .PHONY target as,

A phony target is one that is not really the name of a file; rather it is just a name for a recipe to be executed when you make an explicit request.

However, what does this mean?

What is `.PHONY` in a makefile?

By default, targets are “target files”. For the following rule,

1hello.txt: dependency.txt
2	echo "hello world" > hello.txt

make asks: “Is hello.txt out of date?” To determine if it is out of date, it follows the logic:

If hello.txt exists, is dependency.txt more recent than hello.txt? If so, run the instructions.
If hello.txt does not exist, run the instructions.

1> touch dependency.txt
2> make hello.txt
3echo "hello world" > hello.txt
4> make hello.txt
5make: 'hello.txt' is up to date.
6> echo "update" > dependency.txt
7> make hello.txt
8echo "hello world" > hello.txt

However, for targets that do not output files, a .PHONY label is used. In the following rule,

1.PHONY: format
2format:
3	gofmt -s -w .

the format target formats go code. It does not produce a file.

Why is `.PHONY` important in a makefile?

For some cases, .PHONY is not needed. For example,

1lint:
2	golangci-lint run

the lint target surfaces various code issues. As long as a file named lint does not exist (or will not exist in the future), this command will continue working. From the make perspective, lint will always be out of date since the file will never exist.

The .PHONY is only important if a target’s name can collide with a file name.

List public IP addresses of EC2 instances using AWS CLI

Thomas Taylor — Sat, 25 Mar 2023 00:45:00 -0400

Listing all public IP addresses of EC2 instances can be completed using the --query argument.

1aws ec2 describe-instances \
2  --query "Reservations[*].Instances[*].PublicIpAddress" \
3  --output text

Turn off the pager for AWS CLI

Thomas Taylor — Fri, 24 Mar 2023 20:00:00 -0400

Disabling the AWS CLI pager can be completed in three different ways.

Setting the AWS_PAGER environment variable will disable the pager.

1export AWS_PAGER=
2aws s3 ls

In AWS CLI V2, support for a cli_pager profile option was added. In the ~/.aws/config file, the cli_pager may be used.

1[default]
2cli_pager=

A global argument named --no-cli-pager exists to disable the AWS CLI pager per command usage.

1aws s3 ls --no-cli-pager

Use multiple AWS accounts using AWS CLI

Thomas Taylor — Thu, 23 Mar 2023 00:30:00 -0400

Using multiple AWS accounts from the command line is natively handled with profiles in the AWS CLI.

Using `aws configure`

Using the aws configure command, multiple profiles can be configured.

1aws configure --profile account1
2AWS Access Key ID [None]: ...
3AWS Secret Access Key [None]: ...
4Default region name [None]: ...
5Default output format [None]: ...

Then, the --profile account1 option may be used with future commands.

1aws s3 ls --profile account1

Or an environment variable may be set.

1export AWS_PROFILE=account1
2aws s3 ls # uses account1 credentials

Note: If the profile is named --profile default, it will represent the default profile when no --profile argument is provided.

Manually setting credentials

The ~/.aws/credentials and ~/.aws/config files can be modified directly.

Add the credentials to the ~/.aws/credentials file

1[default]
2aws_access_key_id=accesskey
3aws_secret_access_key=secretaccesskey
4
5[account1]
6aws_access_key_id=accesskey
7aws_secret_access_key=secretaccesskey

Add the profile to the ~/.aws/config file

1[default]
2region=us-east-1
3output=json
4
5[profile account1]
6region=us-east-1
7output=json

Use the --profile argument or set the AWS_PROFILE environment variable.

1aws s3 ls --profile account1

1export AWS_PROFILE=account1
2aws s3 ls # uses account1 credentials

Filter S3 files in bucket by pattern using AWS CLI

Thomas Taylor — Wed, 22 Mar 2023 00:25:00 -0400

Filtering S3 files using a pattern, or finding all files in an S3 bucket that contain a substring can be completed using the AWS CLI.

Filter by S3 object prefix

If the S3 object prefix is predictable, a quick solution is using the native --prefix argument.

1aws s3api list-objects-v2 \
2	--bucket BUCKET_NAME \
3	--prefix PREFIX_STRING

Output:

 1{
 2	"Contents":[
 3		{
 4			"Key":"prefix/",
 5			"LastModified":"2021-11-22T07:51:03+00:00",
 6			"ETag":"...",
 7			"Size":0,
 8			"StorageClass":"STANDARD"
 9		},
10		{
11			"Key":"prefix/example.txt",
12			"LastModified":"2021-12-12T17:18:32+00:00",
13			"ETag":"...",
14			"Size":1646,
15			"StorageClass":"STANDARD"
16		}
17	]
18}

Find S3 objects containing keyword/substring

To find all S3 object keys that contain a certain substring, the --query argument can be utilized.

1aws s3api list-objects-v2 \
2	--bucket BUCKET_NAME \
3	--query "Contents[?contains(Key, `SEARCH_PATTERN`)]"

Output:

 1[
 2	{
 3		"Key":"name-with-substring",
 4		"LastModified":"2021-12-12T17:18:32+00:00",
 5		"ETag":"...",
 6		"Size":1646,
 7		"StorageClass":"STANDARD"
 8	},
 9	{
10		"Key":"another-name-with-substring",
11		"LastModified":"2021-11-22T07:52:45+00:00",
12		"ETag":"...",
13		"Size":1645,
14		"StorageClass":"STANDARD"
15	}
16]

STS assume role in one command using AWS CLI

Thomas Taylor — Tue, 21 Mar 2023 00:00:00 -0400

In order to assume a role, two actions must normally be completed.

How to assume role with STS

1aws sts assume-role \
2	--role-arn ROLE \
3	--role-session-name test

Output:

 1{
 2	"Credentials": {
 3		"AccessKeyId": "...",
 4		"SecretAccessKey": "...",
 5		"SessionToken": "...",
 6		"Expiration": "2023-03-21T11:32:58+00:00"
 7	},
 8	"AssumedRoleUser": {
 9		"AssumedRoleId": "...",
10		"Arn": "..."
11	}
12}

then, export the credentials in the next step

1export AWS_ACCESS_KEY_ID="..."
2export AWS_SECRET_ACCESS_KEY="..."
3export AWS_SESSION_TOKEN="..."

How to assume role in one command

Manually exporting the credentials is a tedious process; however, the command can be simplified to export in one line.

1export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
2$(aws sts assume-role \
3	--role-arn arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME \
4	--role-session-name SESSION_NAME \
5	--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
6	--output text))

This solution uses the printf built-in.

Invalid base64 error Lambda AWS CLI

Thomas Taylor — Mon, 20 Mar 2023 01:20:00 -0400

The Invalid base64 error occurs when invoking a lambda from the AWS CLI because the default --payload parameter expects base64 encoding.

1aws lambda invoke --function-name test --payload '{"movie": "John Wick"}' response.json

Error output:

1Invalid base64: "{"movie": "John Wick"}"

Solve `Invalid base64` lambda invoke error

To solve the Invalid base64 error, supply the --cli-binary-format parameter with the value raw-in-base64-out:

1aws lambda invoke --function-name test --cli-binary-format raw-in-base64-out --payload '{"movie": "John Wick"}' response.json

Output:

1{
2	"StatusCode": 200,
3	"ExecutedVersion": "$LATEST"
4}

Invoke a lambda using a file

A user may opt to supply a .json file:

1{
2	"movie": "John Wick"
3}

1aws lambda invoke --function-name test --cli-binary-format raw-in-base64-out --payload file://request.json response.json

Add new element to JSON array with jq

Thomas Taylor — Sun, 19 Mar 2023 01:15:00 -0400

Adding a new element to an existing JSON array can be completed using jq.

Add element to existing array of objects using `jq`

This is a generic projects.json file:

 1{
 2	"projects": [
 3		{
 4			"id": 1,
 5			"version": "2.3.0"
 6		},
 7		{
 8			"id": 2,
 9			"version": "1.54.0"
10		}
11	]
12}

Appending a new element to the array can be completed in two ways:

1. += []

1cat projects.json | jq '.projects += [{
2	"id": 3,
3	"version": "5.4.1"
4}]'

2. |=

1jq '.projects[.projects | length] = {
2	"id": 3,
3	"version": "5.4.1"
4}'

Output:

 1{
 2	"projects": [
 3		{
 4			"id": 1,
 5			"version": "2.3.0"
 6		},
 7		{
 8			"id": 2,
 9			"version": "1.54.0"
10		},
11		{
12			"id": 3,
13			"version": "5.4.1"
14		}
15	]
16}

Add element to existing array of strings using `jq`

Similarly with strings:

1echo '["id1"]' | jq '. += ["id2"]'

1echo '["id1"]' | jq '.[. | length] = "id2"'

Output:

1[
2	"id1",
3	"id2"
4]

Passing variables to jq

Thomas Taylor — Sat, 18 Mar 2023 01:40:00 -0400

Passing a shell / bash variable / environment variable to jq is simple using the native functionality of:

--arg name value or --argjson name value.

How to pass variable to `jq` filters

This is a generic projects.json file:

 1{
 2	"projects": [
 3		{
 4			"name": "project 1",
 5			"owner": "owner 1",
 6			"id": 1,
 7			"version": "2.3.0"
 8		},
 9		{
10			"name": "project 2",
11			"owner": "owner 1",
12			"id": 2,
13			"version": "1.54.0"
14		},
15		{
16			"name": "project 3",
17			"owner": "owner 2",
18			"id": 3,
19			"version": "4.9.2"
20		}
21	]
22}

String variables

Retrieve project information via a name supplied by a user.

1echo "Enter name: "
2read name
3
4cat projects.json | 
5	jq -r --arg n "$name" '.projects[]|select(.name == $n)'

Output:

1Enter name: 
2project 1
3{
4  "name": "project 1",
5  "owner": "owner 1",
6  "id": 1,
7  "version": "2.3.0"
8}

--arg name value: value is automatically interpreted as a string.

Number variables

Retrieve project information via an id supplied by a user.

1echo "Enter id: "
2read id
3
4cat projects.json | 
5    jq -r --arg id "$id" '.projects[]|select(.id == ($id|tonumber))'

Output:

1Enter id: 
22
3{
4  "name": "project 2",
5  "owner": "owner 1",
6  "id": 2,
7  "version": "1.54.0"
8}

--arg name value: value must be explicitly converted to a number using the native tonumber function.

Optionally, --argjson name value may be used in jq versions 1.5 or higher. --argjson respects the value type. Following the same example as before,

1echo "Enter id: "
2read id
3
4cat projects.json | 
5    jq -r --argjson id "$id" '.projects[]|select(.id == $id)'

Output:

1Enter id: 
22
3{
4  "name": "project 2",
5  "owner": "owner 1",
6  "id": 2,
7  "version": "1.54.0"
8}

How to source shell scripts in Fish

Thomas Taylor — Fri, 17 Mar 2023 01:30:00 -0400

There may be a use case to source a general sh or bash script in fish. For example, if a function is present in a script file that needs to be sourced, fish cannot execute the script because sh functions are not natively supported.

How to source general scripts in `fish`

Sourcing general scripts in fish is trivial with a plugin named bass. Using a preferred package manager, bass can be installed and used to source scripts.

1# variables.sh
2
3example(){
4  export VAR_NAME=test	
5}
6example

1source ./variables.sh

Output:

1from sourcing file ./variables.sh
2source: Error while reading file “./variables.sh”

Using bass:

1bass source ./variables.sh
2echo $VAR_NAME

Output:

1test

Executing a script with dot vs source in Bash

Thomas Taylor — Thu, 16 Mar 2023 01:25:00 -0400

What are the differences between source scriptname.sh, . scriptname.sh and ./scriptname.sh when executing scripts?

Running a program in Bash

Executing a script is simple in Bash and many other shells.

1scriptname.sh

1./scriptname.sh

The shell commands above will execute the scriptname.sh script as a separate program from the command line. It may be any type of script.

Sourcing a script in Bash

When a file is sourced, it runs in the current user’s shell as if the commands were typed out.

1. scriptname.sh

1source scriptname.sh

The most common use case is when scripts export environment variables.

1export ENV_VAR=test

1source test.sh
2echo $ENV_VAR

Output:

1test

Ternary operator in Python

Thomas Taylor — Wed, 15 Mar 2023 01:30:00 -0400

The ternary operator is an in-line if/else statement that many languages support natively; however, Python does not have a specific operator. Instead, it uses conditional expression resolution which was added in Python 2.5.

In-line if/else statement

An in-line if/else statement in Python looks like the following:

1'true' if True else 'false'

Output:

1'true'

Breaking down the syntax,

1true_value if condition else false_value

Return the value of the “idle” stage for a blue/green deployment.

1def get_active_stage():
2    # logic to get the active stage
3    return "blue"
4
5idle_stage = "green" if get_active_stage() == "blue" else "blue"
6print(stage)

Output:

1green

Grep for contents after pattern match

Thomas Taylor — Tue, 14 Mar 2023 01:00:00 -0400

There may be a need to grep for a specific pattern but only capture / echo the contents after the match.

grep will match on any pattern you supply; however, what if a user wants to capture the status of a particular host in the following file?

1hostname1: bad
2hostname2: good
3hostname3: good
4hostname4: bad

Get contents after `grep` match using `sed`

Using a combination of grep and sed, this can be accomplished:

1grep 'hostname1' file.txt | sed 's/^.*: //'

Evaluation:

grep matches any line that contains hostname2
sed replaces (s/// substitute) any character (.*) from the beginning of the line (^) until the last occurrence of the sequence : with the empty string of .

Output:

1bad

Get contents after `grep` match using `awk`

If the delimiter is known ahead of time, awk is a simpler approach.

1grep 'hostname2' file.txt | awk '{print $2}'

Evaluation:

grep matches any line that contains hostname2
awk prints the second field after the delimiter (which is space by default)
1. use the -F option to switch delimiters: ex. awk -F "," for commas

Output:

1good

Get contents using `awk` only

awk may optionally be used by itself to extract the value.

1awk '/hostname4:/ {print $2}' file.txt

Output:

1bad

How to pass variables to Makefiles

Thomas Taylor — Mon, 13 Mar 2023 01:30:00 -0400

Passing variables, exported variables, or .env (dot env) file variables from outside of a Makefile can be completed in a few ways.

Pass variables to Makefile using environment variables

Exported environment variables are referenced by name in the Makefile:

1test:
2	@echo $(TEST)

1export TEST=value
2make test

Output:

1value

Pass variables to Makefile using command line

Similarly, variables may be explicitly passed to make prior to command executions.

1test:
2	@echo $(TEST)

1TEST=value make test

Output:

1value

Pass variables using `.env` (dot env) files

Makefiles can contain an include directive which reads one or more files before executing.

1ifneq (,$(wildcard ./.env))
2	include .env
3	export
4endif
5
6test:
7	@echo $(TEST)

.env file:

1TEST=dotenv

Output:

1dotenv

Write a multi-line command in a Makefile

Thomas Taylor — Sun, 12 Mar 2023 09:45:00 -0400

Occasionally, there is a need for adding multiple continuous lines for one target in a Makefile.

Writing multiple lines for a target

Executing multiple commands in a target is trivial:

1test:
2	@echo 1
3	@echo 2

Output:

11
22

However, each line is executed in its own shell. If a continuous line is needed, the following rules must be applied:

Use $$ instead of $
Terminate each line using a semicolon ;
Add a trailing backslash at the ending of the line \

1test:
2	var1=test; \
3	echo $$var1

Output:

1var1=test; \
2echo $var1
3test

A real use case using multi-line targets

For how.wtf, the CI/CD process invalidates a CloudFront distribution via id.

For context, the aws dependency verifies that aws is executable prior to the command execution. More information on that here.

1.PHONY: invalidate
2invalidate: aws
3	distribution_id=$$(aws cloudfront list-distributions --query "DistributionList.Items[?starts_with(Origins.Items[0].DomainName, '$(BUCKET)')].Id" --output text); \
4	aws cloudfront create-invalidation \
5		--distribution-id $$distribution_id \
6		--paths "/*" \
7		--query "Invalidation.Id" \
8		--output text

1distribution_id=$(aws cloudfront list-distributions --query "DistributionList.Items[?starts_with(Origins.Items[0].DomainName, 'how.wtf')].Id" --output text); \
2aws cloudfront create-invalidation \
3	--distribution-id $distribution_id \
4	--paths "/*" \
5	--query "Invalidation.Id" \
6	--output text
7ID6WOFMWKLCYDYRUA6AGPXXXXX

Writing multiple lines using `ONESHELL`

The ONESHELL directive allows multiple lines to executed in the same shell; however, special characters like @, -, and + are interpreted differently.

1.ONESHELL:
2test:
3	var1=test
4	echo $$var1

Output:

1var1=test
2echo $var1
3test

Check if a program exists from a Makefile

Thomas Taylor — Sat, 11 Mar 2023 00:50:00 -0400

A Makefile can check if a shell program or executable exists before running recipes.

Check if all programs exist

1REQUIRED_BINS := hugo terraform aws exiftool jpegoptim optipng mogrify
2$(foreach bin,$(REQUIRED_BINS),\
3    $(if $(shell command -v $(bin) 2> /dev/null),,$(error Please install `$(bin)`)))

This method will run before any make is completed. It ensures that each program listed in REQUIRED_BINS is executable. If not, it will output an error quickly.

Check if a specific program exists

If it’s preferred to check a shell program before running a specific command, the syntax below will enable for that use-case:

 1tools := hugo terraform aws exiftool jpegoptim optipng mogrify
 2$(tools):
 3	@which $@ > /dev/null
 4	
 5.PHONY: serve
 6serve: hugo
 7	$(HUGO) server -D
 8
 9.PHONY: validate
10validate: terraform
11	$(TERRAFORM) validate

This works by using automatic variables with static patterns. All items in tools will become targets. The $@ in the recipe will be be substituted with the target’s name.

In the example above, make serve will first check if the hugo target is completed. That target runs @which hugo > /dev/null.

Convert OrderedDict to dict in Python

Thomas Taylor — Fri, 10 Mar 2023 00:45:00 -0400

Upon first glance, converting an OrderedDict seems intuitive.

Convert `OrderedDict` to `dict`

1from collections import OrderedDict
2
3od1 = OrderedDict([("key1", "value1"), ("key2", "value2")])
4
5d1 = dict(od1)
6print(d1)

Output:

1{'key1': 'value1', 'key2': 'value2'}

Convert nested `OrderedDict`s to `dict`

However, if an OrderedDict has nested types of OrderedDicts, this method will not convert the nested OrderedDict types to dictionaries:

 1from collections import OrderedDict
 2
 3od1 = OrderedDict(
 4    [
 5        ("key1", "value1"),
 6        ("key2", "value2"),
 7        ("key3", OrderedDict([("key3", "value3")])),
 8    ]
 9)
10
11d1 = dict(od1)
12print(d1)

Output:

1{'key1': 'value1', 'key2': 'value2', 'key3': OrderedDict([('key3', 'value3')])}

Instead, a combination of json.loads() and json.dumps() can be used.

 1from collections import OrderedDict
 2import json
 3
 4od1 = OrderedDict(
 5    [
 6        ("key1", "value1"),
 7        ("key2", "value2"),
 8        ("key3", OrderedDict([("key3", "value3")])),
 9    ]
10)
11
12d1 = json.loads(json.dumps(od1))
13print(d1)

Output:

1{'key1': 'value1', 'key2': 'value2', 'key3': {'key3': 'value3'}}

Set cache control on S3 objects using the AWS CLI

Thomas Taylor — Thu, 09 Mar 2023 08:20:00 -0400

Cache-control can be set on S3 bucket objects.

Setting cache control using `aws s3 cp`

An object’s metadata may only be changed by overwriting it using the metadata-directive flag.

Single file:

1aws s3 cp s3://bucket/example.txt s3://s3-bucket/file.txt \
2  --metadata-directive REPLACE \
3  --cache-control max-age=86400

All files:

1aws s3 cp s3://bucket/ s3://bucket/ \
2  --metadata-directive REPLACE \
3  --cache-control max-age=86400 \
4  --recursive

All image files / specific file extensions

1aws s3 cp s3://bucket/ s3://bucket/ \
2  --metadata-directive REPLACE \
3  --cache-control max-age=86400 \
4  --exclude "*" \
5  --include "*.jpg" \
6  --include "*.gif" \ 
7  --include "*.webp" \
8  --recursive

Setting cache control for websites served by AWS CloudFront

CloudFront is a CDN service provided by AWS. The max-age directive sets the browser caching age in seconds while another directive named s-maxage sets the CloudFront caching in seconds.

Choosing from the examples above, add s-maxage=86400 to the --cache-control flag for all objects in a bucket.

1aws s3 cp s3://bucket/ s3://bucket/ \
2  --metadata-directive REPLACE \
3  --cache-control max-age=86400,s-maxage=86400 \
4  --recursive

Setting cache control using `aws s3 sync`

Warning: Using this option only sets cache headers upon initial sync. The cp command will need to be used to replace headers on existing S3 objects.

1aws s3 sync /path s3://bucket/ \
2  --cache-control max-age=86400,s-maxage=86400

Conditionally add properties to an object in JavaScript

Thomas Taylor — Wed, 08 Mar 2023 08:45:00 -0400

Conditionally adding properties/fields/members to an object in JavaScript or TypeScript can be completed using a combination of the logical and spread operators or simply an if statement.

Add conditional properties using `if`

A simple technique is using an if statement to conditionally add members:

1let obj = {}
2if (condition)
3  obj.field = 5;

There are no thrills to this technique, but it’s straight forward.

Add conditional properties using logical and spread operators

For a shorter syntax, ES5 introduced the ability to do this:

1const obj = {
2  ...(condition && {field: 5})
3}

This technique works due to how logical operators function in JavaScript.

1console.log(true && {field: 5})

The snippet above is processed as follows: starting from left and traversing right, the first operand that is evaluated as falsy is returned. In the expression above, there are no falsy operands so the right-most is returned.

The output:

1{field: 5}

Additionally, the spread operator has a lower precedence than the && operator because the underlying implementation uses Object.assign()

If the condition is false:

1const obj = {
2  val: 2,
3  ...(false && {field: 5})
4}
5console.log(obj)

The spread operation will be result in nothing being added.

1{val:2}

Run workflow, step, or job based on file changes GitHub Actions

Thomas Taylor — Tue, 07 Mar 2023 08:50:00 -0400

There are options for conditionally running GitHub actions at the workflow, job, or step level based on file changes.

Trigger GitHub Action workflow on file changes

GitHub actions natively support triggering workflows based on file changes:

 1on:
 2  push:
 3    paths:
 4    - 'src/**'
 5
 6jobs:
 7  job1:
 8    steps:
 9    - uses: actions/checkout@v3
10    - name: tests
11      run: ...

On a push to any file within the src/** directory, this workflow will execute job1 and its steps.

Run job based on file changes in GitHub Actions

GitHub actions do not natively support triggering jobs based on file changes; however, there is a marketplace action that supports this use-case named dorny/paths-filter.

 1on:
 2  push:
 3    branches:
 4    - main
 5
 6jobs:
 7  changes:
 8    runs-on: ubuntu-latest
 9    outputs:
10      src: ${{ steps.changes.outputs.src }}
11      infra: ${{ steps.changes.outputs.infra }}
12    steps:
13    - uses: actions/checkout@v3
14    - uses: dorny/paths-filter@v2
15      id: changes
16      with:
17        filters: |
18          src:
19            - 'src/**'
20          infra:
21            - 'infra/**'          
22
23  src:
24    needs: changes
25    if: ${{ needs.changes.outputs.src == 'true' }}
26    runs-on: ubuntu-latest
27    steps:
28    - uses: actions/checkout@v3
29    - run: ...
30
31  infra:
32    needs: changes
33    if: ${{ needs.changes.outputs.infra == 'true' }}
34    runs-on: ubuntu-latest
35    steps:
36    - uses: actions/checkout@v3
37    - run: ...

On a push to the main branch, this workflow will execute; however, the first action checks where changes occurred and outputs true or false for the specified filtered paths.

Run step based on file changes in GitHub Actions

Similarly to jobs, GitHub actions to not natively support triggering individual steps based on file changes. dorny/paths-filter supports this use-case as well.

 1on:
 2  push:
 3    branches:
 4    - main
 5
 6jobs:
 7  job1:
 8    steps:
 9    - uses: actions/checkout@v3
10    - uses: dorny/paths-filter@v2
11      id: changes
12      with:
13        filters: |
14          src:
15            - 'src/**'
16          infra:
17            - 'infra/**'
18                      
19    - name: src deploy
20      if: steps.changes.outputs.src == 'true'
21      run: ...
22      
23    - name: infra deploy
24      if: steps.changes.outputs.infra == 'true'
25      run: ...
26    
27    - name: e2e
28      if: steps.changes.outputs.src == 'true' || steps.changes.outputs.infra == 'true'
29      run: ...

Pull requests

On a side note, dorny/paths-filter does not need the checkout action for pull request triggered actions.

 1on:
 2  pull_request:
 3    branches:
 4    - main
 5
 6jobs:
 7  build:
 8    runs-on: ubuntu-latest
 9    # For pull requests, it's not necessary to checkout code
10    permissions:
11      pull-requests: read
12    steps:
13    - uses: dorny/paths-filter@v2
14      id: filter
15      with:
16        filters: ... # Configure your filters

How to unadd or uncommit files in Git

Thomas Taylor — Mon, 06 Mar 2023 00:55:00 -0400

Occasionally, developers will mistakenly git add or git commit files before pushing to remote. Luckily, there are commands that remediate the situation.

How to unadd a file

Unadding a specific file before a commit:

1git reset

How to unadd all files

Unadding all files before a commit:

1git reset

How to uncommit the last commit

If the last commit needs to be uncommitted:

1git reset --soft HEAD~1

The command reads as “undo the last committed changes and preserve the files”. If the files do not need to be preserved, --hard may be used.

At this point, the files may be unadded if desired.

Aliasing

Aliases allow for ease-of-use in daily workflows:

1git config --global alias.unadd 'reset HEAD --'
2git config --global alias.uncommit 'reset --soft HEAD~1'

Usage:

 1> git add example.txt
 2> git status
 3On branch main
 4Your branch is up to date with 'origin/main'.
 5
 6Changes to be committed:
 7  (use "git restore --staged ..." to unstage)
 8	modified:   example.txt
 9
10> git unadd
11Unstaged changes after reset:
12M	example.txt
13> git add example.txt
14> git commit -m "add example.txt"
15[main 1eb1b2f] add example.txt
16 1 file changed, 2 insertions(+), 0 deletions(-)
17 create mode 100644 example.txt
18> git uncommit
19> git unadd
20Unstaged changes after reset:
21M	example.txt

How to remove or delete a PPA in Ubuntu

Thomas Taylor — Sun, 05 Mar 2023 01:15:00 -0400

PPAs (Personal Package Archive) are third-party collections of packages built and maintained by users.

How to list PPAs installed on the system

1ls /etc/apt/sources.list.d/
2neovim-ppa-unstable-jammy.list

How to remove a PPA using terminal

The recommended way to remove a PPA is to use the apt-add-repository command with the --remove (or -r) option.

1sudo apt-add-repository --remove ppa:PPA_REPOSITORY/PPA

Example:

1sudo apt-add-repository -r ppa:neovim-ppa/unstable

How to catch Boto3 errors

Thomas Taylor — Sat, 04 Mar 2023 01:25:00 -0400

Catching errors with the boto3 library is straightfoward.

Catching `boto3` errors with `ClientError`

 1import boto3
 2import botocore
 3
 4s3 = boto3.client("s3")
 5
 6try:
 7    s3.get_object(Bucket="bucket", Key="key")
 8except botocore.exceptions.ClientError as e:
 9    if e.response["Error"]["Code"] == "AccessDenied":
10        print(e.response["Error"]["Message"])
11    else:
12        raise

Other information in the e.response object may additionally be useful:

e.response["Error"]["Code"] for the error code
e.response["Error"]["Message"] for the error message
e.response["ResponseMetadata"]["RequestId"] for the request id
e.response["ResponseMetadata"]["HTTPStatusCode"] for the status code

e.response’s full dictionary:

 1{
 2    'Error': {
 3        'Code': 'AccessDenied',
 4        'Message': 'Access Denied'
 5    }, 
 6    'ResponseMetadata': {
 7        'RequestId': 'requestId',
 8        'HostId': 'hostId',
 9        'HTTPStatusCode': 403,
10        'HTTPHeaders': {
11            'x-amz-request-id': 'requestId',
12            'x-amz-id-2': 'id-2',
13            'content-type': 'application/xml',
14            'transfer-encoding': 'chunked',
15            'date': 'Sat, 04 Mar 2023 06:11:15 GMT',
16            'server': 'AmazonS3'
17        },
18        'RetryAttempts': 0
19    }
20}

Catching `boto3` errors with service exceptions

For some clients, the AWS Python SDK has exposed service exceptions:

 1import boto3
 2import botocore
 3
 4s3 = boto3.client("s3")
 5
 6try:
 7    s3.create_bucket(Bucket="bucket")
 8except s3.exceptions.BucketAlreadyExists:
 9    print("s3 bucket already exists")
10except botocore.exceptions.ClientError as e:
11    print(e)

How to install Microsoft fonts on Ubuntu

Thomas Taylor — Fri, 03 Mar 2023 00:00:00 -0400

Installing Microsoft fonts is simple through the terminal on Debian-based distributions.

Why are Microsoft fonts not installed?

Microsoft fonts like Times New Roman, Arial, etc. are not open source. Because of licensing issues, most linux distributions opt to exclude proprietary software.

Install Microsoft fonts using Terminal / CLI

Add the multiverse repository.

1sudo add-apt-repository multiverse

Install ttf-mscorefonts-installer.

1sudo apt update && sudo apt install ttf-mscorefonts-installer

When the EULA agreement appears, press tab and enter on then on the “Do you accept the EULA license terms?” screen.

Refresh the font cache.

1sudo fc-cache -f -v

If the EULA is mistakenly rejected, reinstall.

1sudo apt install -reinstall ttf-mscorefonts-installer

How to install Microsoft Edge on Ubuntu

Thomas Taylor — Thu, 02 Mar 2023 01:35:00 -0400

Installing Microsoft Edge is simple through the terminal on any Debian-based distribution.

Install Microsoft Edge using Terminal / CLI

Download the Microsoft repository GPG key.

1curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg

Add the GPG key to the trusted keys.

1sudo install -o root -g root -m 644 microsoft.gpg /etc/apt/trusted.gpg.d/

Add the edge repository.

1sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/edge stable main" > /etc/apt/sources.list.d/microsoft-edge-stable.list'

Remove the downloaded GPG key since it was added to the system.

1sudo rm microsoft.gpg

Install Microsoft Edge.

1sudo apt update && sudo apt install microsoft-edge-stable

Disable welcome message in Fish Shell

Thomas Taylor — Wed, 01 Mar 2023 00:45:00 -0400

The default welcome message in Fish can be surpressed or customized.

Disable welcome message

A frequently asked question is “How do I change the greeting message?”. It can be completed by editing the config.fish file:

1vim ~/.config/fish/config.fish

Using the g global flag, the fish_greeting variable can be set to empty.

1set -g fish_greeting

Customize welcome message

In addition to disabling, the message can be customized.

1vim ~/.config/fish/functions/fish_greeting.fish

1function fish_greeting
2    echo "greetings!"
3end

Get size of an S3 bucket with AWS CLI

Thomas Taylor — Tue, 28 Feb 2023 01:15:00 -0400

Using the AWS CLI, S3 bucket sizes can retrieved.

Finding the S3 Bucket Size

Use the s3 list command with the --summarize option.

1aws s3 ls --summarize --human-readable --recursive s3://bucket-name/

Output:

12023-02-27 00:11:33    5.1 KiB object1name
22023-02-06 00:31:42    5.5 KiB object2name
3
4Total Objects: 2
5   Total Size: 10.6 KiB

If the output is too noisy, we can retrieve the last 2 lines of the output:

1aws s3 ls --summarize --human-readable --recursive s3://bucket-name/ | tail -2

Output:

1Total Objects: 2
2   Total Size: 10.6 KiB

Difference between init and new in Python

Thomas Taylor — Mon, 27 Feb 2023 00:05:00 -0400

__init__ and __new__ are dunder methods invoked during the creation of an object instance.

What are the differences?

__new__ is used to control the creation of a new instance
__init__ is used to control the initialization of an instance

`__new__`	`__init__`
Controls the creation of an instance	Controls the initialization of an instance
Invoked first before `__init__`	Invoked after `__new__`
Returns an instance of the class	Returns nothing

__new__ is the first step of an instance creation and is responsible for returning a new instance of the class.

In contrast, the __init__ function initializes the instance after its creation.

Code example

In the code snippet below, __new__ contains the cls class argument while __init__ contains the self reference to the instance.

 1class Example:
 2    def __new__(cls, *args, **kwargs):
 3        print("__new__")
 4        return super().__new__(cls, *args, **kwargs)
 5
 6    def __init__(self):
 7        print("__init__")
 8
 9
10e = Example()

Output:

1__new__
2__init__

Increment an invoked_count variable each time __new__ is called:

 1class Example:
 2    invoked_count = 0
 3
 4    def __new__(cls, *args, **kwargs):
 5        Example.invoked_count += 1
 6        return super().__new__(cls, *args, **kwargs)
 7
 8    def __init__(self):
 9        print(Example.invoked_count)
10
11
12e1 = Example()
13e2 = Example()

Output

11
22

Use cases

__init__ is used as a class constructor for initializing new instances.

A common use case for __new__ is implementing the singleton pattern.

Singleton in Python

Thomas Taylor — Sun, 26 Feb 2023 00:00:00 -0400

The Singleton pattern enforces having one instance of a class. This tutorial will showcase how to implement the singleton pattern in Python.

Implementing the Singleton pattern

Using the __new__ dunder method, the creation of the object can be controlled since __new__ is called before the __init__ method.

 1class Singleton:
 2    _instance = None
 3
 4    def __new__(cls):
 5        if cls._instance is None:
 6            cls._instance = super(Singleton, cls).__new__(cls)
 7        return cls._instance
 8
 9s1 = Singleton()
10s2 = Singleton()
11print(s1 == s2) # outputs True

s1 and s2 are the same object in memory.

Why is this beneficial?

The singleton pattern is useful for certain cases:

Logger
Thread Pooling
Caching
Configuration

Read environment variables from file in Python

Thomas Taylor — Sat, 25 Feb 2023 02:15:00 -0400

A .env file is a text file containing key value pairs of environment variables. This file is normally included with a project, but not committed to source.

How to read `.env` files using Python

Firstly, install the dotenv package.

1pip install python-dotenv

Include the folllowing lines:

1from dotenv import load_dotenv
2
3load_dotenv()

Using load_dotenv(), the application will load the .env file and host environment variables. If this is the .env file for the application:

1VARIABLE1=test

Then the output will be test for the following snippet:

1from dotenv import load_dotenv
2import os
3
4load_dotenv()
5
6print(os.environ.get("VARIABLE1")) # outputs test

However, if the host environment has a VARIABLE1 defined:

1export VARIABLE1=test2

The output will change to test2.

If this is not desired behavior, the author may opt to use dotenv_values which returns a dict of values strictly parsed from the .env file:

1from dotenv import dotenv_values
2
3config = dotenv_values()
4print(config) # outputs OrderedDict([('VARIABLE1', 'test')])

If the .env file resides on a different path than the default root directy of the project, use the dotenv_path option:

1from dotenv import load_dotenv
2from pathlib import Path
3
4load_dotenv(dotenv_path=Path("path/to/file"))

Execute CLI commands in Python

Thomas Taylor — Fri, 24 Feb 2023 23:45:00 -0400

Executing CLI commands / shell scripts / external programs in Python is easy.

Use `subprocess.run`

Since Python 3.5 (subprocess.call for earlier versions), subprocess.run is the recommended way to execute external shell scripts.

1import subprocess
2
3output = subprocess.run(["echo", "hello", "world"], capture_output=True)
4print(output.returncode) # outputs 0
5print(output.stdout.decode("utf-8")) # outputs hello world

Play sounds or music from Linux terminal

Thomas Taylor — Thu, 23 Feb 2023 23:15:00 -0400

Playing music or sounds from the command line is simple with Sound eXchange (SoX). It supports many audio file types: mp3, wav, ogg, etc.

What is `SoX`?

As described by its man page, Sound eXchange is the Swiss Army knife of audio manipulation. It’s a cross platform command line utility used to convert, apply affects, record, or play audio files.

How to play audio from the terminal

Install sox using your package manager with all the encoders.

For Ubuntu:
```
1sudo apt-get update
```
```
1sudo apt-get install sox libsox-fmt-all
```
After the installation is complete, use the play command to play an audio file.
```
1play filename.mp3
```

Difference between continue and pass in Python

Thomas Taylor — Wed, 22 Feb 2023 00:10:00 -0400

Using the `continue` statement

A continue statement is used within a for loop to skip the remainder of the current iteration. Essentially, the continue statement will simply skip to the next iteration.

A normal for loop that prints the current index i:

1for i in range(3):
2    print(i)

10
21
32

The same for loop, with a conditional and a continue:

1for i in range(3):
2    if i == 1:
3        continue
4    print(i)

10
22

On the second iteration of the for loop (when i = 1), the continue statement stopped the current iteration and “continued” to the next.

Using the `pass` statement

As implied by the name, the pass state does nothing.

1for i in range(3):
2    if i == 1:
3        pass
4    print(i)

10
21
32

This is most useful when defining placeholders for future code to satisfy the interpreter.

1def func():
2    pass

Static methods in Python

Thomas Taylor — Tue, 21 Feb 2023 00:40:00 -0400

Static methods are similar to instance methods (class-level methods), except static methods behave like regular functions bound to a class namespace. Unlike instance methods, static methods do not have access to cls or self.

Creating static methods

A class named StringUtil contains a is_empty method for determining if a string is empty.

1class StringUtil:
2    @staticmethod
3    def is_empty(s):
4        return not s.strip()
5
6print(StringUtil.is_empty("test")) # False
7print(StringUtil.is_empty(" ")) # True

When to use static methods

Static methods are advantageous for:

Singletons
Utility classes with similar grouped methods
Methods that belong in a class namespace, but do not require instance variables (self)

How to delete files and directories in Python

Thomas Taylor — Mon, 20 Feb 2023 00:40:00 -0400

How to delete a file or folder using Python?

Removing files

Remove a file with `os.remove()`

1import os
2
3os.remove("path/to/file")

Remove a file with `pathlib.Path.unlink()`

1from pathlib import Path
2
3p = Path("path/to/file")
4p.unlink()

Removing directories

Remove an empty directory with `os.rmdir()`

1import os
2
3os.rmdir("path/to/folder")

Remove an empty directory with `pathlib.Path.rmdir()`

1from pathlib import Path
2
3p = Path("path/to/folder")
4p.rmdir()

Remove an empty directory and its contents with `shutil.rmtree()`

1import shutil
2
3shutil.rmtree("path/to/folder")

Python dataclasses with optional fields

Thomas Taylor — Sun, 19 Feb 2023 01:20:00 -0400

How do I create dataclasses in Python with optional fields?

How does a dataclass work?

Dataclasses generate dunder methods on the user’s behalf. Instead of defining __init__, __eq__, etc. manually, these methods are provided out of the box simply be defining fields and their respestive types.

1from dataclasses import dataclass
2
3@dataclass
4class Person
5    name: str
6    age: int

Because the methods and constructor are generated, the attributes are checked and hardcoded.

Optional fields in dataclasses

There are options for making an argument optional with dataclasses.

Using `Optional[str]` and a default value

It is possible to make an optional argument by using a default value for the attribute.

 1from dataclasses import dataclass
 2from typing import Optional
 3
 4@dataclass
 5class Person
 6    name: str
 7    age: int
 8    email: Optional[str] = None
 9
10print(Person(name="john", age=35)) # Person(name='john', age=35, email=None)

Using `InitVar`

InitVar is passed to the __init__ and __post_init__ methods, but is not stored as a class attribute. This can be valuable for some use-cases.

If a movie is not published, mark it as hidden.

 1from dataclasses import dataclass, field, InitVar
 2from typing import Optional
 3
 4@dataclass
 5class Movie:
 6    title: str
 7    hidden: bool = field(init=False)
 8    is_published: InitVar[Optional[bool]] = True
 9
10    def __post_init__(self, is_published):
11        self.hidden = not(is_published)
12
13print(Movie(title="title", is_published=False)) # Movie(title='title', hidden=True)
14print(Movie(title="title", is_published=True)) # Movie(title='title', hidden=False)
15print(Movie(title="title")) # Movie(title='title', hidden=False)

CloudFormation create stack vs deploy in AWS CLI

Thomas Taylor — Sat, 18 Feb 2023 02:30:00 -0400

CloudFormation provides two options for deploying templates using the AWS CLI:

aws cloudformation create-stack
aws cloudformation deploy

What are the differences between the two?

`create-stack`

Create stack is a specific API action for creating AWS CloudFormation stacks.

1aws cloudformation create-stack \
2    --stack-name STACK_NAME \
3    --template-body file://path/to/file.json

`deploy`

Deploy is an abstraction for managing AWS CloudFormation stacks and change sets. The list of API actions available for CloudFormation does not include deploy.

Unlike create-stack, which is a direct API call, deploy combines create-change-set and execute-change-set in a single convenient command.

The following command will either create a new stack if one does not exist, or create a change set and execute the change set to update an existing stack.

1aws cloudformation deploy \
2    --stack-name STACK_NAME \
3    --template-body file://path/to/file.json

If the author wants to review the change set before automatically applying, the flag --no-execute-changeset can be used:

1aws cloudformation deploy \
2    --stack-name STACK_NAME \
3    --template-body file://path/to/file.json \
4    --no-execute-changeset

Should I use `create-stack` or `deploy`?

In general, aws cloudformation deploy will be easiest to use. It automatically handles creating or updating change sets on the user’s behalf.

How to view or tail AWS logs in real time

Thomas Taylor — Fri, 17 Feb 2023 00:10:00 -0400

Since mid-2019, the AWS CLI natively supports real time viewing of aws logs.

View latest logs using `--since`

View the latest 365 days worth of logs:

1aws logs tail $GROUP --since 365d

View the latest 52 weeks worth of logs:

1aws logs tail $GROUP --since 52w

View other supported units here.

Real time follow logs using `--follow`

Continuously poll for new logs using --follow:

1aws logs tail $GROUP --follow

Other options with `aws logs tail`

For other supported options with aws logs tail, view the full reference here.

How to copy files in Linux

Thomas Taylor — Thu, 16 Feb 2023 00:40:00 -0400

Copying files and directories is an essential task in Linux.

The `cp` command

The cp command is used to copy files and directories from one location to another.

Basic syntax:

1Usage: cp [OPTION]... [-T] SOURCE DEST
2  or:  cp [OPTION]... SOURCE... DIRECTORY
3  or:  cp [OPTION]... -t DIRECTORY SOURCE...
4Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.

The SOURCE can be one or many files / directories, and the DEST can be a single file or directory.

Copying multiple files to a directory

1cp file1 file2 file3 dir1

Copying matching files to a directory

1cp *.txt dir1

Copying a directory to another directory

Using the -r or -R or --recursive option, cp will recursively copy a directory - including its files and subdirectories.

1cp -r dir1 ~/Documents

Copying files without clobbering

Using the -i or --interactive option, cp will iteratively request user input for files with naming conflicts.

1> cp -i file1 dir1
2cp: overwrite 'dir1/file1'?

Include verbosity when copying files

1> cp -v file1 dir1
2'file1' -> 'dir1/file1'

How to move files in Linux

Thomas Taylor — Wed, 15 Feb 2023 00:25:00 -0400

Moving files is an essential task individuals often need to perform.

The `mv` command and its usage

The mv command is used to rename or move files and directories from one location to another.

Basic syntax:

1> mv --help
2Usage: mv [OPTION]... [-T] SOURCE DEST
3  or:  mv [OPTION]... SOURCE... DIRECTORY
4  or:  mv [OPTION]... -t DIRECTORY SOURCE...`

The SOURCE can be one or many files / directories, and the DEST can be a single file or directory.

Moving multiple files to a directory

1mv file1 file2 dir1

Moving matching files to a directory

1mv *.md dir1

Moving a directory to another directory

1mv dir1 ~/Documents

Moving files without clobbering

Using the -i or --interactive option, mv will iteratively request user input for files with naming conflicts.

1> mv -i file1 ~/Documents
2mv: overwrite '~/Documents/file1'?

Renaming files

Rename file1 to file2:

1mv file1 file2

Create multiple directories at once in terminal

Thomas Taylor — Tue, 14 Feb 2023 00:00:00 -0400

Here are different strategies for creating multiple directories/folders or subdirectories at once in a linux terminal.

Using `mkdir` positional arguments

mkdir can be supplied multiple positional arguments.

1mkdir dir1 dir2

Using `bash` brace expansion + `mkdir`

mkdir can be supplied multiple positional arguments using bash’s brace expansion.

Creating directories

1mkdir {dir1,dir2}

Creating subdirectories

-p flag is used to create the parent directory if it does not already exist.

1mkdir -p dir/{dir1,dir2}

Creating directories with `bash` sequences

1mkdir dir{1..5}

Adding a new IAM User using the AWS CLI

Thomas Taylor — Mon, 13 Feb 2023 00:20:00 -0400

How to create a new IAM user with the AWS CLI?

Create a new IAM User

To create a new IAM user, issue the following command:

1aws iam create-user --user-name USER

1{
2    "User": {
3        "Path": "/",
4        "UserName": "USER",
5        "UserId": "AIDA...",
6        "Arn": "arn:aws:iam::123456789012:user/USER",
7        "CreateDate": "2023-02-13T05:03:54+00:00"
8    }
9}

Add IAM User to an IAM Group

Create IAM Group

If an IAM Group does not exist, create one.

1aws iam create-group --group-name GROUP

1{
2    "Group": {
3        "Path": "/",
4        "GroupName": "GROUP",
5        "GroupId": "AGPA...",
6        "Arn": "arn:aws:iam::123456789012:group/GROUP",
7        "CreateDate": "2023-02-13T05:05:45+00:00"
8    }
9}

Grant access to IAM Group

The managed policy arn arn:aws:iam::aws:policy/PowerUserAccess is attached to the group in this example.

1aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/PowerUserAccess --group-name GROUP

Add the user to the IAM group.

1aws iam add-user-to-group --user-name USER --group-name GROUP

Verify user is added to group

1aws iam get-group --group-name GROUP --query "Users[*].{name:UserName,id:UserId}"

1[
2    {
3        "name": "USER",
4        "id": "AIDA..."
5    }
6]

Grant AWS Console Access

1aws iam create-login-profile --user-name USER --password PASSWORD --no-password-reset-required

1{
2    "LoginProfile": {
3        "UserName": "USER",
4        "CreateDate": "2023-02-13T05:10:17+00:00",
5        "PasswordResetRequired": false
6    }
7}

Grant Programmatic Access

1aws iam create-access-key --user-name USER

1{
2    "AccessKey": {
3        "UserName": "USER",
4        "AccessKeyId": "AKIA...",
5        "Status": "Active",
6        "SecretAccessKey": "UU...",
7        "CreateDate": "2023-02-13T05:11:18+00:00"
8    }
9}

Clear all CloudFront cache and files using AWS CLI

Thomas Taylor — Sun, 12 Feb 2023 14:30:00 -0400

How to invalidate all files or clear all cache for CloudFront using the AWS CLI?

Finding the distribution id

If the distribution id is unknown, issue the following command to list all distribution ids:

1aws cloudfront list-distributions --query "DistributionList.Items[*].Id"

Clear or Invalidate all cache for CoudFront

To invalidate all paths, run the create-invalidation command with a wildcard path /*.

1aws cloudfront create-invalidation --distribution-id CF_DISTRIBUTION_ID --paths "/*"

 1{
 2    "Location": "https://cloudfront.amazonaws.com/2020-05-31/distribution/CF_DISTRIBUTION_ID/invalidation/I3SD80NE98KDJA",
 3    "Invalidation": {
 4        "Id": "I3SD80NE98KDJA",
 5        "Status": "InProgress",
 6        "CreateTime": "2023-02-12T19:20:31.982000+00:00",
 7        "InvalidationBatch": {
 8            "Paths": {
 9                "Quantity": 1,
10                "Items": [
11                    "/*"
12                ]
13            },
14            "CallerReference": "cli-1676229631-287576"
15        }
16    }
17}

Clear specific files from Cache

If one or more files need invalidation, but not the entire distribution, simply execute the following command:

1aws cloudfront create-invalidation --distribution-id CF_DISTRIBUTION_ID \
2    --paths "/path/of/file" "/path/to/other/file" "/path/to/folder/*"

Notice the wildcard for /path/to/folder/*. This allows any files under /path/to/folder/ to be invalidated with the request.

Check progress on validation

To check the progress on an invalidation request, issue the follow command:

1aws cloudfront get-invalidation --id INVALIDATION_ID --distribution-id CF_DISTRIBUTION_ID

 1{
 2    "Invalidation": {
 3        "Id": "I3SD80NE98KDJA",
 4        "Status": "Completed",
 5        "CreateTime": "2023-02-12T19:20:31.982000+00:00",
 6        "InvalidationBatch": {
 7            "Paths": {
 8                "Quantity": 1,
 9                "Items": [
10                    "/*"
11                ]
12            },
13            "CallerReference": "cli-1676229631-287576"
14        }
15    }
16}

Find AWS Account Id using CLI or Boto3

Thomas Taylor — Sat, 11 Feb 2023 01:15:00 -0400

How to find the AWS Account ID using the AWS CLI or Boto3?

AWS CLI

Retrieve your full identity

1aws sts get-caller-identity

1{
2    "UserId": "AIDA...",
3    "Account": "123456789012",
4    "Arn": "arn:aws:iam::123456789012:user/user"
5}

Use native AWS CLI --query option for querying the output.

1aws sts get-caller-identity --query "Account" --output text

1123456789012

Python Boto3

1import boto3
2
3sts = boto3.client("sts")
4account_id = sts.get_caller_identity()["Account"]
5print(account_id) # 0123456789012

How to get start or end of a day in Python

Thomas Taylor — Fri, 10 Feb 2023 00:00:00 -0400

In Python, datetime natively handles getting the earliest/latest moment of a given day using the combine function.

Get the start of a day

1from datetime import datetime, time
2
3start_of_day = datetime.combine(datetime.now(), time.min)
4print(start_of_day) # 2023-02-10 00:00:00

Get the end of a day

1from datetime import datetime, time
2
3end_of_day = datetime.combine(datetime.now(), time.max)
4print(end_of_day) # 2023-02-10 23:59:59.999999

Resource vs Client in Python Boto3

Thomas Taylor — Thu, 09 Feb 2023 00:45:00 -0400

Resource and Client are two different abstractions in the Boto3 SDK. What are the differences between the two?

Clients

Clients provide lower-level access to AWS Services. They typically represent a 1:1 functionality mapping of any AWS service API and are generated via JSON service definition files such as the one for DynamoDB. The botocore library is shared between the AWS CLI & Boto3.

For example, the DynamoDB PutItem API call requires TableName and Item parameters to be supplied. The Item parameter is required to be a PutItemAttributeMap which forces the user to add typing for each value. In addition to handling the strict input types, users are required to handle pagination.

1import boto3
2
3dynamodb = boto3.Client("dynamodb")
4dynamodb.put_item(
5    TableName="table_name",
6    Item={
7        "id": { "S": "test" }
8    }
9)

Resources

Resources provide higher-level access to AWS Services and reside within boto3. They are only supported for a subset of AWS services and are generated via JSON resource definition files such as the one for DynamoDB

For example, the DynamoDB PutItem API call is completed using the Table object.

1import boto3
2
3dynamodb = boto3.Resource("dynamodb")
4table = dynamodb.Table("table_name")
5table.put_item(
6    Item={
7        "id": "test"
8    }
9)

As opposed to the clients, resources typically handle marshalling data and pagination.

Should I use a client or resource?

If resources are available for the service your application is interfacing with, use them. Otherwise, use clients. The full list of services are here.

Note

Per the boto3 resources documentation,

The AWS Python SDK team does not intend to add new features to the resources interface in boto3. Existing interfaces will continue to operate during boto3’s lifecycle. Customers can find access to newer service features through the client interface.

The AWS Python SDK team will not be adding new resource abstractions; however, the resources that are supported will function in perpetuity.

DynamoDB error: key element does not match the schema

Thomas Taylor — Wed, 08 Feb 2023 00:00:00 -0400

The provided key element does not match the schema occurs when getting or deleting a record without providing all the proper key elements.

How to fix the error

If your DynamoDB schema includes a partition/hash key and a range/sort key, you must specify both attributes.

Given the schema:

1{
2  "pk": "test_pk",
3  "sk": "test_sk"
4}

Python Boto3:

 1import boto3
 2
 3dynamodb = boto3.resource("dynamodb")
 4table = dynamodb.Table("test")
 5table.get_item(
 6    Key={
 7        "pk": "test_pk",
 8        "sk": "test_sk"
 9    }
10)
11
12table.delete_item(
13    Key={
14        "pk": "test_pk",
15        "sk": "test_sk",
16        "other": "test"
17    }
18)

You must specify a region error AWS

Thomas Taylor — Tue, 07 Feb 2023 00:00:00 -0400

You must specify a region error occurs when a default AWS Region is not configured.

Error:

1You must specify a region. You can also configure your region by running "aws configure"

How to fix the `You must specify a region` error

Method 1: Set the AWS_DEFAULT_REGION environment variable

1export AWS_DEFAULT_REGION="us-east-1"

Method 2: Include the default region in the ~/.aws/config file or run aws configure

1> aws configure
2  AWS Access Key ID [None]:
3  AWS Secret Access Key [None]:
4  Default region name [None]: us-east-1
5  Default output format [None]:

The ~/.aws/config file will be generated with the following configuration:

1[default]
2region = us-east-1

Method 3: Explicity specify the region with the AWS API

AWS CLI:

1aws ec2 describe-instances --region us-east-1

Python Boto3:

1import boto3
2
3kms = boto3.client("kms", region_name="us-east-1")

JavaScript AWS SDK

1import { S3Client } from '@aws-sdk/client-s3';
2
3const s3Client = new S3Client({
4  region: 'us-east-1'
5});

Storing large items in DynamoDB

Thomas Taylor — Mon, 06 Feb 2023 00:30:00 -0400

Each item in a DynamoDB table has a maximum size limit of 400 KB, including both the attribute names and values. This limit applies to all data types: strings, numbers, and binary data.

The three best ways to mitigate the maximum size limit:

Partitioning the data
Compressing the data
Storing data in S3

How to handle large data in DynamoDB

Partition the data

A simple way to get around the item size limit is to split the data into multiple items.

Table Name: lorem

pk	sk	data
lorem	p#0	Lorem ipsum dolor sit…
lorem	p#1	Euismod nisi porta lo…
lorem	p#2	rcu risus quis varius…
lorem	p#3	phasellus. Enim praes…

 1import boto3
 2
 3def partition_data(data, size):
 4    return [data[i:i+size] for i in range(0, len(data), size)]
 5
 6
 7# 100 paragraphs of Lorem ipsum
 8lorem = """
 9Lorem ipsum dolor sit amet, consectetur adipiscing elit,
10sed do eiusmod tempor incididunt ut labore et dolore magna 
11aliqua. Sem integer vitae justo eget magna. At tellus at 
12urna condimentum mattis pellentesque id. Habitasse...
13"""
14
15dynamodb = boto3.resource("dynamodb")
16table = dynamodb.Table("lorem")
17partition_key = "lorem"
18sort_key_prefix = "p#" # p for partition
19
20# Write chunks to DynamoDB
21chunks = partition_data(lorem, 5000)
22for i, c in enumerate(chunks):
23    table.put_item(
24        Item={
25            "pk": partition_key,
26            "sk": f"{sort_key_prefix}{i}",
27            "data": c
28        }
29    )
30
31# Read chunks from DynamoDB
32response = table.query(
33    KeyConditionExpression="pk = :pk and begins_with(sk, :sk)",
34    ExpressionAttributeValues={
35        ":pk": partition_key,
36        ":sk": sort_key_prefix
37    },
38    ScanIndexForward=True
39)
40
41# Query for all paginated results if applicable.
42items = response["Items"]
43while "LastEvaluatedKey" in response:
44    response = table.query(ExclusiveStartKey=response["LastEvaluatedKey"])
45    items.update(response["Items"])
46
47# Concatenate the data field from all the items
48lorem_from_dynamodb = "".join(i["data"] for i in items)
49
50print(lorem == lorem_from_dynamodb) # prints True

Compress the data

Try to reduce the size of your data by compression. Compression algorithms like Gzip can significantly reduce the size of the data.

Compressing a single item

Table Name: lorem

pk	data
lorem	eJy1Xdly3LquffdX6ANu+…
lorem	eJytXFl227gS/c8quALtw…

 1import boto3
 2import zlib
 3
 4
 5def compress_data(data):
 6    return zlib.compress(data.encode())
 7
 8
 9# 100 paragraphs of Lorem ipsum
10lorem = """
11Lorem ipsum dolor sit amet, consectetur adipiscing elit,
12sed do eiusmod tempor incididunt ut labore et dolore magna 
13aliqua. Sem integer vitae justo eget magna. At tellus at 
14urna condimentum mattis pellentesque id. Habitasse...
15"""
16
17dynamodb = boto3.resource("dynamodb")
18table = dynamodb.Table("lorem")
19partition_key = "lorem"
20sort_key = "lorem"
21
22table.put_item(Item={"pk": partition_key, "data": compress_data(lorem)})
23
24response = table.get_item(Key={"pk": partition_key})
25data = response["Item"]["data"]
26
27lorem_from_dynamodb = zlib.decompress(bytes(data)).decode()
28print(lorem_from_dynamodb == lorem)  # prints true

Compressing a partitioned item

DynamoDB schema:

Table Name: lorem

pk	sk	data
lorem	p#0	eJy1Xdly3LquffdX6ANu+…
lorem	p#1	eJytXFl227gS/c8quALtw…

 1import boto3
 2import zlib
 3
 4
 5def partition_data(data, size):
 6    return [data[i : i + size] for i in range(0, len(data), size)]
 7
 8
 9def compress_data(data):
10    return zlib.compress(data.encode())
11
12
13# 100 paragraphs of Lorem ipsum
14lorem = """
15Lorem ipsum dolor sit amet, consectetur adipiscing elit,
16sed do eiusmod tempor incididunt ut labore et dolore magna 
17aliqua. Sem integer vitae justo eget magna. At tellus at 
18urna condimentum mattis pellentesque id. Habitasse...
19"""
20
21dynamodb = boto3.resource("dynamodb")
22table = dynamodb.Table("lorem")
23partition_key = "lorem"
24sort_key_prefix = "p#"  # p for partition
25
26# Write chunks to DynamoDB
27chunks = partition_data(lorem, 50000)
28for i, c in enumerate(chunks):
29    table.put_item(
30        Item={
31            "pk": partition_key,
32            "sk": f"{sort_key_prefix}{i}",
33            "data": compress_data(c),
34        }
35    )
36
37# Read chunks from DynamoDB
38response = table.query(
39    KeyConditionExpression="pk = :pk and begins_with(sk, :sk)",
40    ExpressionAttributeValues={":pk": partition_key, ":sk": sort_key_prefix},
41    ScanIndexForward=True,
42)
43
44# Query for all paginated results if applicable.
45items = response["Items"]
46while "LastEvaluatedKey" in response:
47    response = table.query(ExclusiveStartKey=response["LastEvaluatedKey"])
48    items.update(response["Items"])
49
50# Concatenate the data field from all the items
51lorem_from_dynamodb = "".join(
52    zlib.decompress(bytes(i["data"])).decode() for i in items
53)
54
55print(lorem_from_dynamodb == lorem) # prints true

Store the data in S3

Consider storing the data in S3 as opposed to an attribute value in DynamoDB.

Table Name: lorem

pk	s3_key
lorem	s3://bucket/key

 1import boto3
 2
 3# 100 paragraphs of Lorem ipsum
 4lorem = """
 5Lorem ipsum dolor sit amet, consectetur adipiscing elit,
 6sed do eiusmod tempor incididunt ut labore et dolore magna 
 7aliqua. Sem integer vitae justo eget magna. At tellus at 
 8urna condimentum mattis pellentesque id. Habitasse...
 9"""
10
11bucket_name = "bucket_name"
12object_key = "object_key"
13partition_key = "lorem"
14s3_key = f"s3://{bucket_name}/{object_key}"
15
16# Store data in S3 object
17s3 = boto3.client("s3")
18s3.put_object(Bucket=bucket_name, Key=object_key, Body=lorem.encode())
19
20# Store reference to S3 object in DynamoDB
21dynamodb = boto3.resource("dynamodb")
22table = dynamodb.Table("lorem")
23table.put_item(Item={"pk": partition_key, "s3_key": s3_key})
24
25# Get reference to S3 object in DynamoDB
26response = table.get_item(Key={"pk": partition_key})
27s3_key = response["Item"]["s3_key"]
28
29# Read contents of S3 object
30bucket, key = s3_key[5:].split("/")  # remove "s3://" prefix and split on "/"
31response = s3.get_object(Bucket=bucket, Key=key)
32lorem_from_s3 = response["Body"].read().decode()
33
34print(lorem_from_s3 == lorem) # prints True

How to push an empty commit in Git

Thomas Taylor — Thu, 01 Dec 2022 23:05:00 -0400

Pushing an empty commit using Git is straightforward. Git provides an optional flag on the git commit command: --allow-empty.

Pushing without making changes

1git commit --allow-empty -m "empty commit"

1git push

The commit will be pushed without requiring changes.

Why is this useful?

This is useful for retrying a CI pipeline when an issue occurs with a GitHub webhook or CI service.

Interfaces in Python

Thomas Taylor — Sat, 01 Oct 2022 15:30:00 -0400

Python does not have the traditional interface keyword as seen in other programming languages. Instead, the language supports ducktyping and multiple inheritance which may satisfy the need for interfaces for some use cases.

However, there are native pythonic methods to create / implement an interface.

Ducktyping

If it quacks like a duck and walks like a duck, Python assumes it’s a duck:

 1from dataclasses import dataclass
 2
 3@dataclass
 4class User:
 5    id: str
 6    name: str
 7
 8class UserService:
 9    def get_user_by_id(self, id: str) -> User:
10        """Retrieve user by ID"""
11        pass
12
13class Database(UserService):
14    def get_user_by_id(self, id: str) -> User:
15        """Retrieves user by ID from an database"""
16        # Implementation goes here
17        pass
18
19class Cache(UserService):
20    def get_user_by_id(self, id: str) -> User:
21        """Retrieves user by ID from cache"""
22        # Implementation goes here
23        pass

Due to the lack of enforcement, this paradigm may not scale well for the implementer’s needs.

Abstract Base Classes

Introduced in Python 2.6, abstract base classes allow for interface creation.

 1from abc import ABC, abstractmethod
 2from dataclasses import dataclass
 3
 4@dataclass
 5class User:
 6    id: str
 7    name: str
 8
 9class UserService(ABC):
10    @abstractmethod
11    def get_user_by_id(self, id: str) -> User:
12        """Retrieve user by ID"""
13        raise NotImplementedError
14
15class Database(UserService):
16    def get_user_by_id(self, id: str) -> User:
17        """Retrieves user by ID from a database"""
18        # Implementation goes here
19        pass

This covers two cases:

The inheriting class must implement the abstract method
The implementation cannot instantiate this parent’s method: super().get_user_by_id(id)

How to Set Default Audio in Ubuntu

Thomas Taylor — Mon, 27 Jun 2022 02:10:00 -0400

Using the pactl command, setting default audio in Ubuntu is straightforward. For startup persistence, Ubuntu Startup Applications comes in handy.

pactl is the cli entry for the PulseAudio sound server and will be used in this tutorial.

List all input and output audio

List audio outputs:

1pactl list short sinks

11	alsa_output.pci-0000_06_00.6.HiFi__hw_Generic__sink	module-alsa-card.c	s16le 2ch 48000Hz	RUNNING

List audio inputs:

1pactl list short sources

11	alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_797_2019_02_24_60522-00.analog-stereo	module-alsa-card.c	s16le 2ch 44100Hz	SUSPENDED
22	alsa_input.pci-0000_06_00.6.HiFi__hw_Generic__source	module-alsa-card.c	s16le 2ch 44100Hz	SUSPENDED
33	alsa_input.pci-0000_06_00.6.HiFi__hw_acp__source	module-alsa-card.c	s32le 2ch 48000Hz	SUSPENDED

Take note of the devices listed in the output.

Set default input and output audio

Command to set default audio output:

1pactl set-default-sink

Example usage:

1pactl set-default-sink alsa_output.pci-0000_06_00.6.HiFi__hw_Generic__sink

Command to set default audio input:

1pactl set-default-source

Example usage:

1pactl set-default-source alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_797_2019_02_24_60522-00.analog-stereo

Persist command on start/restart

Persisting audio output:

Open the application named “Startup Applications” (should be preinstalled)
Click “Add” button in the upper right hand corner
Enter a command name such as: “Set default audio output” in the Name: field
Place the pactl set-default-sink command in the Command: field
Click “Add” button in the bottom right hand corner of the prompt

Persisting audio input:

Click “Add” button in the upper right hand corner
Enter a command name such as: “Set default audio input” in the Name: field
Place the pactl set-default-source command in the Command: field
Click “Add” button in the bottom right hand corner of the prompt

Check if a String Contains a Substring in Python

Thomas Taylor — Fri, 24 Jun 2022 01:30:00 -0400

Checking if a string contains a substring is trivial in Python!

The `in` Operator

The in operator is the most straightforward and pythonic method.

Checking if a string has a substring:

1string = "howwtf"
2substring = "wtf"
3
4if substring in string:
5    print("exists")

1exists

Checking if a string has any substrings:

1string = "howwtf"
2substrings = ["how", "apple"]
3if any(x in string for x in substrings):
4    print("exists")

1exists

Checking if a string has all substrings:

1string = "howwtf"
2substrings = ["how", "wtf"]
3if all(x in string for x in substrings):
4    print("all exist")

1all exist

For Loops in Bash

Thomas Taylor — Sun, 19 Jun 2022 22:30:00 -0400

Using a for loop in Bash is simple!

for loop over strings

1for i in "follow" "the" "white" "rabbit"; do
2    echo "${i}"
3done

1follow
2the
3white
4rabbit

for loop with range

1for i in {0..2}; do
2    echo "${i}"
3done

10
21
32

bash 4 added an increment parameter for ranges

1for i in {0..6..2}; do
2    echo "${i}"
3done

for loop with array

1a=("fried" "green" "tomatoes")
2for i in ${a[@]}; do
3    echo "${i}"
4done

1fried
2green
3tomatoes

for loop C-style

1for (( i = 0; i <= 3; i++ )); do
2    echo "${i}"
3done

for loop control statements: break & continue

break statement

The break statement terminates the current loop.

1for i in "first" "second" "break" "third"; do
2    if [[ "${i}" == "break" ]]; then
3        break
4    fi
5    echo "${i}"
6done

1first
2second

continue statement

The continue statement skips the current iteration and ‘continues’ to the next iteration.

1for i in "first" "second" "break" "third"; do
2    if [[ "${i}" == "break" ]]; then
3        continue
4    fi
5    echo "${i}"
6done

1first
2second
3third

Deploy AWS Cloudfront and security headers with Terraform

Thomas Taylor — Sun, 12 Dec 2021 02:15:00 -0500

I made a post in July 2021 regarding AWS CloudFront + security headers. Using AWS Cloudfront and Cloudfront Functions, security headers were injected in viewer responses; however, AWS Cloudfront natively supports security headers as of Nov. 2nd, 2021 alongside configurable CORS and custom HTTP response headers.

Security Headers

Cloudfront can natively support all the security headers from the last post:

permissions-policy
referrer-policy
strict-transport-security
x-content-type-options
x-frame-options
x-xss-protection

Terraform Code

Begin with defining an aws_cloudfront_response_headers_policy resource in Terraform. This resources contains all the header policy information. In the following example, the values for each security_headers_config were copied from AWS’s documentation.

 1resource "aws_cloudfront_response_headers_policy" "headers_policy" {
 2  name = "security-headers-policy"
 3
 4  custom_headers_config {
 5    items {
 6      header = "permissions-policy"
 7      override = true
 8      value = "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()"
 9    }
10  }
11
12  security_headers_config {
13    content_type_options {
14      override = true
15    }
16
17    frame_options {
18      override = true
19      frame_option = "DENY"
20    }
21
22    referrer_policy {
23      override = true
24      referrer_policy = "same-origin"
25    }
26
27    strict_transport_security {
28      override = true
29      access_control_max_age_sec = 63072000
30      include_subdomains = true
31      preload = true
32    }
33
34    xss_protection {
35      override = true
36      mode_block = true
37      protection = true
38    }
39  }
40}

Attach it to a Cloudfront Behavior. In this case, I attached it to the default_cache_behavior block.

 1locals {
 2  cloudfront_origin_id = "s3"
 3}
 4
 5resource "aws_cloudfront_distribution" "distribution" {
 6  aliases = var.domain_names
 7  enabled = true
 8
 9  origin {
10    domain_name = aws_s3_bucket.bucket.bucket_domain_name
11    origin_id = local.cloudfront_origin_id
12  }
13
14  default_cache_behavior {
15    allowed_methods = ["GET", "HEAD"]
16    cached_methods = ["GET", "HEAD"]
17    compress = true
18    target_origin_id = local.cloudfront_origin_id
19    viewer_protocol_policy = "redirect-to-https"
20    response_headers_policy_id = aws_cloudfront_response_headers_policy.headers_policy.id
21  }
22
23  restrictions {
24    geo_restriction {
25      restriction_type = "none"
26    }
27  }
28}

Once your infrastructure is deployed, AWS CloudFront will forward security headers. To see this in action, refer to the how.wtf repository.

How to split a string in Bash

Thomas Taylor — Sat, 11 Dec 2021 02:30:00 -0500

Splitting a string in bash is easy with the help of the internal field separator (IFS).

Bash string split using IFS

1string="you got a friend in me"
2IFS=' ' read -ra split <<< "$string"
3echo "${split[*]}"
4# Output: you got a friend in me
5echo "${split[3]}"
6# Output: friend

The above method does not interfere with the IFS global variable since it’s only set for that single invocation.

How to upgrade the AWS CDK CLI

Thomas Taylor — Sat, 28 Aug 2021 01:10:00 -0400

AWS CDK CLI users are commonly faced with an error:

This CDK CLI is no compatible with the CDK library used by your application. Please upgrade the CLI to the latest version

Luckily, there are two easy methods for resolving that involve a single command.

NPM

npm install -g aws-cdk@latest

Yarn

yarn global upgrade aws-cdk

Using Virtualenv with Python

Thomas Taylor — Sat, 10 Jul 2021 21:15:00 -0400

What is Virtualenv?

Virtualenv is a tool used for creating isolated Python environments. It allows multiple Python environments/projects to coexist on the same computer with their own dependencies, python versions, and pip versions.

How to install Virtualenv

Using Python’s package manager, pip or pip3, virtualenv can be installed with a single command:

1pip install --user virtualenv

Optionally, you can install it for all users:

1pip install virtualenv

How do I create a new virtualenv environment?

Simply navigate to the project directory and use the virtualenv cli command with a name for the environment (venv in this case):

1$ cd ~/projects/myproject
2$ virtualenv venv
3created virtual environment CPython3.8.5.final.0-64 in 1993ms
4  creator CPython3Posix(dest=/home/user/projects/myproject, clear=False, no_vcs_ignore=False, global=False)
5  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/user/.local/share/virtualenv)
6    added seed packages: pip==21.0.1, setuptools==56.0.0, wheel==0.36.2
7  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

This creates a new folder named venv in the project directory.

How do I use/activate a virtualenv (venv) environment?

To activate the virtualized environment, run the following command from the project directory:

On Linux / MacOS:

1source venv/bin/activate

On Windows:

1.\venv\Scripts\activate

With the isolated environment activated, any pip install packagename will be contained within the directories of the virtualized environment.

How do I deactivate a virtualenv (venv) environment?

Simply type the following command to deactivate:

1deactivate

Deploy Cloudfront Functions to add security headers with AWS CDK

Thomas Taylor — Sun, 04 Jul 2021 09:15:00 -0400

With the newly published CloudFront Functions, developers can leverage fast and short lived functions to handle simplistic tasks for viewer requests and responses.

The AWS article covers the differences between Lambda@Edge and CloudFront Functions in detail.

For a quick reference, here is the table from it:

Attribute	CloudFront Functions	Lambda@Edge
Runtime support	JavaScript (ECMAScript 5.1 compliant)	Node.js / Python
Execution location	218+ CloudFront Edge Locations	13 CloudFront Regional Edge Caches
CloudFront triggers supported	Viewer request / Viewer response	Viewer request /Viewer response / Origin request / Origin response
Maximum execution time	Less than 1 millisecond	5 seconds (viewer triggers) 30 seconds (origin triggers)
Maximum memory	2MB	128MB (viewer triggers) / 10GB (origin triggers)
Total package size	10 KB	1 MB (viewer triggers) / 50 MB (origin triggers)
Network access	No	Yes
File system access	No	Yes
Access to the request body	No	Yes
Pricing	Free tier available; charged per request	No free tier; charged per request and function duration

The Cloudfront Function Code

Using Javascript (ECMAScript 5.1 compliant), the following code adds common security headers to viewer responses:

permissions-policy
referrer-policy
strict-transport-security
x-content-type-options
x-frame-options
x-xss-protection

Create a new file named headers.js:

 1function handler(event) {
 2    var response = event.response
 3    var headers = response.headers;
 4
 5    headers['permissions-policy'] = {
 6        value: 'accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()',
 7    }
 8    headers['referrer-policy'] = { value: 'same-origin'}; 
 9    headers['strict-transport-security'] = { value: 'max-age=63072000; includeSubdomains; preload' };
10    headers['x-content-type-options'] = { value: 'nosniff' }; 
11    headers['x-frame-options'] = { value: 'DENY' }; 
12    headers['x-xss-protection'] = { value: '1; mode=block' }; 
13
14    return response;
15};

This code was a modified version from the AWS documentation described here.

The AWS CDK Code

The production version of the code in this section can be found in the how.wtf open source repository.

Create a requirements.txt

For this tutorial, version 1.111.0 of the AWS CDK was used.

1aws-cdk.aws_cloudfront===1.111.0
2aws-cdk.aws_s3===1.111.0
3aws-cdk.core===1.111.0

Install the dependencies

1pip3 install -r requirements.txt

Create app.py

1from aws_cdk import core
2from stack import WebsiteStack
3
4app = core.App()
5WebsiteStack(app, "website")
6
7app.synth()

Create cdk.json

1{
2    "app": "python3 app.py"
3}

Create stack.py with a basic S3 Bucket

 1from aws_cdk import core
 2from aws_cdk import aws_cloudfront as cloudfront
 3from aws_cdk import aws_s3 as s3
 4
 5class WebsiteStack(core.Stack):
 6
 7    def __init__(self, app: core.App, id: str) -> None:
 8        super().__init__(app, id)
 9
10        bucket = s3.Bucket(
11            self,
12            "bucket",
13            website_index_document="index.html",
14            public_read_access=True,
15            removal_policy=core.RemovalPolicy.DESTROY,
16        )

Add the CloudFront Function + Distribution

 1from aws_cdk import core
 2from aws_cdk import aws_cloudfront as cloudfront
 3from aws_cdk import aws_s3 as s3
 4
 5class WebsiteStack(core.Stack):
 6
 7    def __init__(self, app, id):
 8        super().__init__(app, id)
 9
10        bucket = s3.Bucket(
11            self,
12            "bucket",
13            website_index_document="index.html",
14            public_read_access=True,
15            removal_policy=core.RemovalPolicy.DESTROY,
16        )
17
18        security_headers = cloudfront.Function(
19            self,
20            "security_headers",
21            code=cloudfront.FunctionCode.from_file(
22                file_path="headers.js",
23            ),
24        )
25
26        distribution = cloudfront.CloudFrontWebDistribution(
27            self,
28            "cdn",
29            origin_configs=[
30                cloudfront.SourceConfiguration(
31                    s3_origin_source=cloudfront.S3OriginConfig(
32                        s3_bucket_source=bucket,
33                    ),
34                    behaviors=[
35                        cloudfront.Behavior(is_default_behavior=True),
36                        cloudfront.Behavior(
37                            path_pattern="*",
38                            function_associations=[
39                                cloudfront.FunctionAssociation(
40                                    event_type=cloudfront.FunctionEventType.VIEWER_RESPONSE,
41                                    function=security_headers,
42                                ),
43                            ],
44                        ),
45                    ],
46                )
47            ],
48        )
49
50        core.CfnOutput(
51            self,
52            "distribution-domain-name",
53            value=distribution.distribution_domain_name,
54        )

The directory structure should look like this:

project/
├── app.py
├── cdk.json
├── headers.js
├── requirements.txt
└── stack.py

Deploy the stack

1cdk deploy website

Because of the CfnOutput, the distribution’s domain name is exposed via an output on the stack:

Outputs:
website.distributiondomainname = hostname.cloudfront.net

Add index.html document to the S3 bucket

1<h1>Security headers!h1>

Test the headers

After adding the index.html document, visit the distribution’s domain name to ensure it is working correctly.

To test the security headers, either use your favorite request tool or use securityheaders.com.

Convert two lists into a dictionary in Python

Thomas Taylor — Sat, 22 May 2021 13:35:00 -0400

Converting two lists into a dictionary is a breeze in Python when using the dict and zip methods! Additionally, there are alternatives.

Using Zip & Dict

1>>> students = ["Cody", "Ashley", "Kerry"]
2>>> grades = [93.5, 95.4, 82.8]
3>>> dict(zip(students, grades))
4{'Cody': 93.5, 'Ashley': 95.4, 'Kerry': 82.8}

Dictionary Comprehension

1>>> students = ["Cody", "Ashley", "Kerry"]
2>>> grades = [93.5, 95.4, 82.8]
3>>> {s: g for s, g in zip(students, grades)}
4{'Cody': 93.5, 'Ashley': 95.4, 'Kerry': 82.8}

Python <= 2.6

1>>> from itertools import izip
2>>> students = ["Cody", "Ashley", "Kerry"]
3>>> grades = [93.5, 95.4, 82.8]
4>>> dict(izip(keys, values))
5{'Cody': 93.5, 'Ashley': 95.4, 'Kerry': 82.8}

Sort a list of tuples by an item in Python

Thomas Taylor — Sat, 08 May 2021 23:00:00 -0400

Python has native sort capabilities available for use. You can sort a list of objects by leveraging one of the method specified below.

The questions that this post addresses:

Q: How do you sort a list of tuples by the 1st item?
Q: How do you sort a list of tuples by the 2nd item?
Q: How do you sort a list of tuples in reverse?

`sorted()` function

The sorted() function returns a sorted iterable object, such as a list, in a specified order. It does not modify the list in place.

By default, the sorted() function evaluates the items to determine their order. In the example below, the lexical ordering of the 1st index determines the sorted list.

1lst = [("val2", 2), ("val1", 1)]
2print(sorted(lst))
3# Output: [('val1', 1), ('val2', 2)]

If the 1st elements are identical, the 2nd element determines the ordering:

1lst = [("val1", 2), ("val1", 1)]
2print(sorted(lst))
3# Output: [('val1', 1), ('val2', 2)]

You can optionally specify an index to sort on using a lambda function:

1lst = [("val2", 2), ("val1", 1)]
2print(sorted(lst, key=lambda x: x[1]))
3# Output: [('val1', 1), ('val2', 2)]

Reverse as well:

1lst = [("val2", 2), ("val1", 1)]
2print(sorted(lst, key=lambda x: x[1], reverse=True))
3# Output: [('val2', 2), ('val1', 1)]

Using itemgetter instead of lambda:

1from operator import itemgetter
2lst = [("val1", 1), ("val2", 2)]
3print(sorted(lst, key=itemgetter(1), reverse=True))
4# Output: [('val2', 2), ('val1', 1)]

`list.sort()` function

The list.sort() method sorts a list in place and does not return a value.

Similar to the sorted() function, it uses the lexical ordering of the 1st index to determine the sorted list.

1lst = [("val2", 2), ("val1", 1)]
2lst.sort()
3print(lst)
4# Output: [('val1', 1), ('val2', 2)]

If the 1st elements are identical, the 2nd element determines the ordering:

1lst = [("val1", 2), ("val1", 1)]
2lst.sort()
3print(lst)
4# Output: [('val1', 1), ('val2', 2)]

You can optionally specify an index to sort on using a lambda function:

1lst = [("val2", 2), ("val1", 1)]
2lst.sort(key= lambda x: x[1])
3print(lst)
4# Output: [('val1', 1), ('val2', 2)]

Reverse as well:

1lst = [("val1", 1), ("val2", 2)]
2lst.sort(key= lambda x: x[1], reverse=True)
3print(lst)
4# Output: [('val2', 2), ('val1', 1)]

Using itemgetter instead of lambda:

1from operator import itemgetter
2lst = [("val1", 1), ("val2", 2)]
3lst.sort(key=itemgetter(1), reverse=True)
4print(lst)
5# Output: [('val2', 2), ('val1', 1)]

Ternary operator in Bash

Thomas Taylor — Sun, 02 May 2021 01:35:00 -0400

The ternary operator is a form of syntactic sugar for an in-line if/else statement. Many languages natively support the operator:

JavaScript:

1const color = 'blue';
2console.log(color === 'blue' ? '🟦' : '🟩');
3// Output: 🟦

Python:

1color = "blue"
2print("🟦" if color == "blue" else "🟩")
3# Output: 🟦

1#include 
2#include 
3
4int main()
5{
6    char color[4] = "blue";
7    printf("%s", (strcmp(color, "blue") == 0) ? "🟦" : "🟩");
8}
9// Output: 🟦

Ternary Operation

Bash does not have a native ternary operator. Instead, the same functionality can be achieved using:

1color="blue"
2[[ "$color" == "blue" ]] && echo "🟦" || echo "🟩"
3# Output: 🟦

[[ "$color" == "blue" ]] has an exit status of 0, so it evaluates the right expression of the && and echoes the blue emoji.

Similarly, if color="green", the [[ "$color" == "blue" ]] expression would have a nonzero exit status, so it evaluates the next logical statement (||) and echos the green emoji.

Saving to a variable

1color="blue"
2emoji=$([[ "$color" == "blue" ]] && echo "🟦" || echo "🟩")
3echo $emoji
4# Output: 🟦

⚠️ Caution

If the right-hand side of the && condition has a nonzero exit status, it will silently default to the or (||) expression.

[[ cond ]] && op1 || op2 ➜ op2 will be selected if op1 fails.

Takeaway: Be mindful of the op1 operation. Ensure it exits with a 0 status code, or you may receive a false negative.

Spread operator in Python

Thomas Taylor — Wed, 28 Apr 2021 01:00:00 -0400

The JavaScript spread operator (...) is a useful and convenient syntax for expanding iterable objects into function arguments, array literals, or other object literals.

Python contains a similar “spread” operator that allows for iterable unpacking. Each of the examples below will demonstrate the comparison between the two languages.

Function Arguments

JavaScript:

1function multiply(a, b) {
2    return a * b;
3}
4const numbers = [3, 5];
5console.log(multiply(...numbers));
6// Output: 15

Python:

1def multiply(a, b):
2    return a * b
3numbers = [3, 5]
4print(multiply(*numbers))
5# Output: 15

Array Literals

1const numbers = [1, 2, 3];
2const newNumbers = [0, ...numbers, 4]
3console.log(newNumbers);
4// Output: [ 0, 1, 2, 3, 4 ]

1numbers = [1, 2, 3]
2new_numbers = [0, *numbers, 4]
3print(new_numbers)
4# Output: [0, 1, 2, 3, 4]

Object Literals

1const testObj = { foo: 'bar' };
2console.log({ ...testObj, foo2: 'bar2' });
3// Output: { foo: 'bar', foo2: 'bar2' }

A very similar technique can be applied with Python dictionaries. Notice the double asterisk operator (**).

1test_obj = { 'foo': 'bar' }
2print({ **test_obj, 'foo2': 'bar2' })
3# Output: {'foo': 'bar', 'foo2': 'bar2'}

To unpack keyword arguments, the double asterisk operator (**) is used. In contrast, the single asterisk operator (*) is used for iterable objects.

AWS CLI query examples

Thomas Taylor — Sat, 24 Apr 2021 15:30:00 -0400

While some users may prefer to pipe AWS CLI JSON output to jq for parsing, it is possible to leverage the --query functionality that’s built-in.

Commonly, users deal with large JSON outputs when executing AWS CLI commands in their environments. To mitigate the process, JMESPath, a powerful filtering language, can specify resource properties, create lists, search output, etc.

AWS CLI Output

For reference, the AWS CLI documentation lists JSON document outputs. The commands listed below use aws ec2 describe-images, but any combination of the examples can be used for other services and properties.

1aws ec2 describe-images \
2    --region us-east-1 \
3    --image-ids ami-1234567890EXAMPLE

Output (provided by AWS):

 1{
 2    "Images": [
 3        {
 4            "VirtualizationType": "hvm",
 5            "Description": "Provided by Red Hat, Inc.",
 6            "PlatformDetails": "Red Hat Enterprise Linux",
 7            "EnaSupport": true,
 8            "Hypervisor": "xen",
 9            "State": "available",
10            "SriovNetSupport": "simple",
11            "ImageId": "ami-1234567890EXAMPLE",
12            "UsageOperation": "RunInstances:0010",
13            "BlockDeviceMappings": [
14                {
15                    "DeviceName": "/dev/sda1",
16                    "Ebs": {
17                        "SnapshotId": "snap-111222333444aaabb",
18                        "DeleteOnTermination": true,
19                        "VolumeType": "gp2",
20                        "VolumeSize": 10,
21                        "Encrypted": false
22                    }
23                }
24            ],
25            "Architecture": "x86_64",
26            "ImageLocation": "123456789012/RHEL-8.0.0_HVM-20190618-x86_64-1-Hourly2-GP2",
27            "RootDeviceType": "ebs",
28            "OwnerId": "123456789012",
29            "RootDeviceName": "/dev/sda1",
30            "CreationDate": "2019-05-10T13:17:12.000Z",
31            "Public": true,
32            "ImageType": "machine",
33            "Name": "RHEL-8.0.0_HVM-20190618-x86_64-1-Hourly2-GP2"
34        }
35    ]
36}

Please note Images is a top-level list-type element in the JSON document.

Examples

Warning: The AWS describe-images commands outputs a large JSON document. The JMESPath list slicing (Images[:3]) feature is leveraged to truncate the results. If the full output is desired, use Images[] instead.

Listing Outputs

List the ImageId of the first 3 images owned by Amazon:

1$ aws ec2 describe-images \
2  --owner amazon \
3  --query "Images[:3].ImageId"
4[
5    "aki-04206613",
6    "aki-0a4aa863",
7    "aki-12f0127b"
8]

Note: Images[:3].ImageId outputs the property into a single list. If a list for each image is desired, use Images[:3].[ImageId]. As shown in the next example, it’s useful for listing multiple properties.

List the ImageId and OwnerId of the first 3 images owned by Amazon:

 1$ aws ec2 describe-images \
 2  --owner amazon \
 3  --query "Images[:3].[ImageId,OwnerId]"
 4[
 5    [
 6        "aki-04206613",
 7        "137112412989"
 8    ],
 9    [
10        "aki-0a4aa863",
11        "470254534024"
12    ],
13    [
14        "aki-12f0127b",
15        "470254534024"
16    ]
17]

List the ImageId and OwnerId of the first 3 images owned by Amazon in a list of JSON objects:

If desired, the output values can be nested in a hash object using JMESPath’s hash multi-selection feature.

 1$ aws ec2 describe-images \
 2  --owner amazon \
 3  --query "Images[:3].{image:ImageId,owner:OwnerId}"
 4[
 5    {
 6        "image": "aki-04206613",
 7        "owner": "137112412989"
 8    },
 9    {
10        "image": "aki-0a4aa863",
11        "owner": "470254534024"
12    },
13    {
14        "image": "aki-12f0127b",
15        "owner": "470254534024"
16    }
17]

Filter Projection & Pipe Expressions

List the ImageId of the first 3 images that are owned by Amazon’s OwnerId of 137112412989:

JMESPath provides two features needed to accomplish this query:

1$ aws ec2 describe-images \
2  --owner amazon \
3  --query "Images[?OwnerId=='137112412989'] | [:3].ImageId"
4[
5    "aki-04206613",
6    "aki-499ccb20",
7    "aki-5c21674b"
8]

The combination of a filter projection, Images[?OwnerId=='137112412989'], and a pipe expression, | [:3].ImageId satisfies the requirement.

Here’s the Python code representing the filter projection operation:

1result = []
2for image in inputData['Images']:
3  if image['OwnerId'] == '137112412989'
4    result.append(image)
5return result

It returns:

 1[
 2    {
 3        "ImageId": "aki-04206613",
 4        "OwnerId": "137112412989",
 5        "..."
 6    },
 7    {
 8        "ImageId": "aki-499ccb20",
 9        "OwnerId": "137112412989",
10        "..."
11    },
12    {
13        "ImageId": "aki-5c21674b",
14        "OwnerId": "137112412989",
15        "..."
16    },
17    "...",
18    "...",
19    "..."
20]

Then, grab the first three using the pipe expression:

| [:3]

 1[
 2    {
 3        "ImageId": "aki-04206613",
 4        "OwnerId": "137112412989",
 5        "..."
 6    },
 7    {
 8        "ImageId": "aki-499ccb20",
 9        "OwnerId": "137112412989",
10        "..."
11    },
12    {
13        "ImageId": "aki-5c21674b",
14        "OwnerId": "137112412989",
15        "..."
16    }
17]

Then, add the specific ImageId property:

| [:3].ImageId

1[
2    "aki-04206613",
3    "aki-499ccb20",
4    "aki-5c21674b"
5]

Function Expressions

JMESPath supports function expressions:

List the Id and CreationDate of the first 3 images that are owned by Amazon whose PlatformDetails contain the string ‘Linux’ in sorted order by CreationDate:

Step #1: Use the contains function and grab the first 3 results:

 1$ aws ec2 describe-images \
 2  --owner amazon \
 3  --query "Images[?contains(PlatformDetails, 'Linux')] | [:3]"
 4[
 5    {
 6        "CreationDate": "2016-09-28T21:31:10.000Z",
 7        "ImageId": "aki-04206613",
 8        "PlatformDetails": "Linux/UNIX",
 9        "..."
10    },
11    {
12        "CreationDate": "2009-12-15T18:44:15.000Z",
13        "ImageId": "aki-0a4aa863",
14        "PlatformDetails": "Linux/UNIX",
15        "..."
16    },
17    {
18        "CreationDate": "",
19        "ImageId": "aki-12f0127b",
20        "PlatformDetails": "Linux/UNIX",
21        "..."
22    }
23]

This filters the top-level Images for all objects with Linux in their PlatformDetails value.

Step #2: Sort by CreationDate:

1$ aws ec2 describe-images \
2  --owner amazon \
3  --query "Images[?contains(PlatformDetails, 'Linux')] | [:3] | sort_by(@, &CreationDate)[].{CreationDate:CreationDate,Id:ImageId}"

The sort_by function requires an input similar to this:

1KeyNameHere: [
2    {
3        "key1": "val1",
4        "key2": "val2"
5    }
6]

If results were ordered by the key2 field, the query would be: sort_by(KeyNameHere, &key2)

Similarly, the input to the sort_by function in the example was:

 1[
 2    {
 3        "CreationDate": "2016-09-28T21:31:10.000Z",
 4        "ImageId": "aki-04206613",
 5        "PlatformDetails": "Linux/UNIX",
 6        "..."
 7    },
 8    { "..." },
 9    { "..." }
10]

There is not a top-level key present. In this case, the character @ is used to represent the list that is being passed. Additionally, the key that is being sorted on must begin with an &.

After the sort_by function, the list is “retrieved” [] and the result is formatted to a customized JSON output.

 1[
 2    {
 3        "CreationDate": "",
 4        "Id": "aki-12f0127b"
 5    },
 6    {
 7        "CreationDate": "2009-12-15T18:44:15.000Z",
 8        "Id": "aki-0a4aa863"
 9    },
10    {
11        "CreationDate": "2016-09-28T21:31:10.000Z",
12        "Id": "aki-04206613"
13    }
14]

The first object contains a null CreationDate value. Luckily, it can be removed from the output using the not_null function.

 1$ aws ec2 describe-images \
 2  --owner amazon \
 3  --query "Images[?contains(PlatformDetails, 'Linux')] | [:4] | @[?not_null(CreationDate)] | sort_by(@, &CreationDate)[].{CreationDate:CreationDate,Id:ImageId}"
 4[
 5    {
 6        "CreationDate": "2009-12-15T18:44:15.000Z",
 7        "Id": "aki-0a4aa863"
 8    },
 9    {
10        "CreationDate": "2010-09-28T03:52:22.000Z",
11        "Id": "aki-1c669375"
12    },
13    {
14        "CreationDate": "2016-09-28T21:31:10.000Z",
15        "Id": "aki-04206613"
16    }
17]

Lastly, a common scenario is to filter based on environment variables:

Retrieve a cloudformation export by name and store the value

1$ export_name=$(aws cloudformation list-exports \
2    --query "Exports[?Name=='$ENV_VAR'].Value" \
3    --output text)
4$ echo $export_name
5value_here

The environment variable was used by wrapping with single quotes!

3 ways to format strings in Python

Thomas Taylor — Wed, 21 Apr 2021 01:25:00 -0400

In Python versions 3.6 and above, literal string interpolation receives the hype; however, did you know that there are several ways to format strings in Python?

String Formatting (printf-style)

String objects have a built-in operation: the % operator (modulo).

If the operator is used once in a string, then a single non-tuple object may be used:

1>>> "Ogres are like %s" % "onions"
2'Ogres are like onions'

In this example, the conversion type, %s (string), will be replaced by “onions”.

Otherwise, the strings must be placed within a tuple of exact length or a dictionary.

Tuple:

1>>> "%s are like %s" % ("Ogres", "onions")
2'Ogres are like onions'

Dictionary:

1>>> "%(character)s are like %(vegetable)s" % {"character": "Ogres", "vegetable": "onions"}
2'Ogres are like onions'

Additionally, there are flag characters and other types. To represent an integer, the type %d (for decimal) could be used. In the example below, %d was used in combination with the 0 flag character to add trailing zeros.

1>>> "%03d - License to kill" % 7
2'007 - License to kill'

More information can be found here about the printf-style string formatting.

String Format Method

While the predecessor leveraged type conversions and flag characters, Python 3 introduced the str.format(*args, **kwargs) string method.

Using positional arguments:

1>>> "{}, {}, {}".format("one", "two", "three")
2'one, two, three'

Using indices:

1>>> "{2}, {1}, {0}".format("apple", "orange", "cow")
2'cow, orange, apple'

Using repeated indices:

1>>> "{1}, it's {0}. Why {0}?".format(13, 'Naturally')
2"Naturally, it's 13. Why 13?"

Named arguments:

1>>> "{character} are like {food}".format(character="Ogres", food="onions")
2'Ogres are like onions'

Accessing arguments’ items:

1>>> coord = (1, 5)
2>>> "Plot the coordinate: ({0[0]}, {0[1]})".format(coord)
3'Plot the coordinate: (1, 5)'

Thousands separator:

1>>> "It's over {:,}!".format(9000)
2"It's over 9,000!"

… and more format string syntax tricks here.

Literal String Interpolation

Python 3.6 introduced literal string interpolation, also known as f-strings. F-strings allow you to embed expressions inside string constants.

1>>> pet_name = "Toto"
2>>> state = "Kansas"
3>>> f"{pet_name}, I've a feeling we're not in {state} anymore."
4"Toto, I've a feeling we're not in Kansas anymore"

F-strings are prefixed with the letter f and allow you to complete operations inline within the string.

1>>> f"3 + 5 = {3+5}"
2'3 + 5 = 8'

The existing syntax from the str.format() method can be used.

1>>> f"It's over {9000:,}!"
2"It's over 9,000!"