Author: rchapin

VS Code “Test result not found for:” When Running Tests for a Python Project [SOLVED]

Posted on by rchapin

I finally was able to get Visual Studio Code set-up correctly to run and debug unit and integration tests for a Python 3.8 project that I am working on (I’ll add a link to that post here once it is up).

After making some changes to the code and adding a test I got the following error when trying to debug the test:

Test result not found for: ./mylibs/integration_tests/myclient_integration_test.py::MyClientIntegrationTest::test_happy_path

? An odd error message, to be sure.

After a little while I figured out that when this happens it is ultimately the result of some syntax, interpretation error that occurs at runtime that the IDE may not flag as a problem for you.

Check the Output panel and click on the drop-down and select Python Test Log to see the stack trace of the error to see where you have a typo.

List the Roles for a User or Service Account in a Specific GCP Project

Posted on by rchapin

If you do not have web console permissions to do so, but have the ability to activate a service account that has the viewer permissions or IAM permissons to list IAM roles in a given project, the following is how you can list the roles for a given user or service account.

gcloud projects get-iam-policy <gcp-project> \
--filter="bindings.members:<email-address>" \
--flatten="bindings[].members" --format="table(bindings.role)"

Compiling Python Under Linux

Posted on by rchapin

The following should work with just about any version of Python. I am using it to compile, currently 3.10.x, on distros where those packages are not readily available for installation. The following is a quick how to on getting it compiled under both RedHat/CentOS/Almalinux and Debian based systems.

Download the Tarball for the Version You Want To Install

Download the tar.gz archive for the version that you want to install from here. Verify the download and then save the path to this file for later.

Install Dependencies

This assumes that you already have the “build-essentials” and kernel headers installed on the box, which is an exercise for the reader.

RedHat/CentOS/Almalinux

yum install -y bzip2-devel expat-devel gdbm-devel ncurses-devel openssl-devel readline-devel wget sqlite-devel tk-devel xz-devel zlib-devel libffi-devel gmp-devel libmpc-devel mpfr-devel openssl-devel liblzma-devel

Debian

apt install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libsqlite3-dev libreadline-dev libffi-dev curl libbz2-dev liblzma-dev

Compile Python

The following enables a non-root user to unpack, compile, and install it into their home directory. Copy this file to /var/tmp/compile-python.sh and then run as follows

/var/tmp/compile-python.sh <path-to-tarball>
#!/bin/bash

set -u
set -e

# The path to the downloaded tarball
py_tarball=$1

export PY_DIR=$(echo $py_tarball | awk -F/ '{ print $NF }' | sed 's/.tgz//')
export PY_PREFIX=$(echo ~/usr/local/$PY_DIR | tr [:upper:] [:lower:])

mkdir -p ~/usr/local/src ~/usr/local/bin ~/usr/local/include $PY_PREFIX
rm -rf $PY_PREFIX
tar -xzf $py_tarball -C ~/usr/local/src/
cd ~/usr/local/src/$PY_DIR
./configure --prefix=$PY_PREFIX --exec-prefix=$PY_PREFIX
make && make install

Add the following to your PATH in ~/.bash_profile

PYTHON_HOME=~/usr/local/python-<version>

export PATH=$PATH:$PYTHON_HOME/bin

Using fc to Edit and Re-execute Bash Commands

Posted on by rchapin

I recently learned about the Bash built-in fc. It is a great tool that enables you to edit and re-execute commands from your bash history.

Oftentimes there is a command in your history that instead of just grepping through the history and then re-executing as-is you’ll want to make a modification or two. With fc you can first edit it in your favorite editor and then when closing the editor fc will execute the command.

For me, vim is my editor of choice. Add the following to your .bashrc and fc will automatically open vim for you.

export FCEDIT=vim

Then, simply run fc passing it the id of the command in your history that you want to edit and then execute.

fc 1234

Fluffy Waffles

Posted on by rchapin

Ingredients

  • 1 3/4 cups Flour
  • 2 teaspoons baking powder
  • 1/2 teaspoon salt
  • 1 tablespoon sugar or 1 tablespoon honey
  • 3 eggs – separated
  • 1 1/2 cup milk
  • 1/3 cup vegetable oil
  • 1/4 tsp vanilla

Instructions

  1. Sift dry ingredients in a bowl
  2. Add egg yolks, milk, vegetable oil, and vanilla to dry ingredients – using a mixer, beat thoroughly
  3. Beat egg whites until stiff and gently fold egg whites into batter
  4. For use in waffle iron – Follow manufacturer directions! Makes 3 – 4 round 9″ waffles

Black Bean Cupcakes

Posted on by rchapin

Makes about 12 cupcakes.

Ingredients

Cupcakes

  • 1/2 cups black beans cooked and drained
  • 3 large eggs
  • 1 cup plus 2 tablespoons granulated sugar
  • 4 tablespoons unsalted melted butter
  • 1/2 cups cocoa powder
  • 1 teaspoon instant cofee
  • 1 teaspoon vanilla extract
  • 1/2 teaspoon baking soda
  • 1/2 teaspoon salt
  • 1/3 cup of chopped dark chocolate

Glaze

  • 4 tablespoons unsalted butter
  • 1 tablespoons cocoa powder
  • 1/2 cup powdered sugar

Instructions

Cupcakes

  1. Preheat seep to 350F
  2. In a blender add all at the ingredients except the chocolate and blend until smooth
  3. Add the dark chocolate and blend for about 15 seconds to bleak up the pieces
  4. Pour the batter into the cupcake liners filling each about 3/4 full
  5. Bake for about 20 minutes or until the cupcakes spring back to the touch
  6. Let them cool for about 10 minutes before removing from pan and glazing

Glaze

  1. In a small saucepan over medium-low heat add all of the ingredients and whisk until butter is melted and the mixture is smooth
  2. Remove from heat and allow to cool slightly
  3. Dip the lops of the cupcakes the glaze

Creating a Launch Config in VSCode to Debug a Python Invoke Script

Posted on by rchapin

I regularly use Python Invoke and Fabric for the automation of various tasks; from deploying code to developing my own set of tools for various projects. Following is an example on how to write a launch.json launch configuration for vscode so that you can step through the tasks.py code and debug it.

Assuming that you have created a virtual environment and pip installed invoke into it. And, assuming that you have defined a task in your tasks.py file as follows:

from invoke import task

@task()
def do_something(ctx, some_path, some_other_path):
    # Do something with data in these dirs . . .

The following is a template you can use for a launch configuration that you can use to debug your task. 

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "invoke",
      "type": "python",
      "request": "launch",
      // The complete path to the invoke python script in your virtual environment
      "program": "/my/virtualenv/path/bin/invoke",
      "justMyCode": false,
      // The args that you would otherwise enter on the command line
      // when invoking your task
      "args": [
        "do-something",
        "--some-path",
        "/var/tmp/a/",
        "--some-other-path",
        "/var/tmp/b/"
      ],
      "cwd": "/the/path/to/the/dir/that/contains/your/tasks/script",
    }
  ]
}

Using bq load Command to Load logicalType Partitioned Data into a BigQuery Table

Posted on by rchapin

Following is the syntax and bq load command that you need to issue if you want to load data in avro file into a partitioned BigQuery table based on avro field defined as a logicalType.

Given the following schema

{
  "type" : "record",
  "name" : "logicalType",
  "namespace" : "com.ryanchapin.tests",
  "fields" : [ {
    "name" : "id",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "value",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "day",
    "type" : {
      "type" : "int",
      "logicalType" : "date"
    }
  }
}

And the following BigQuery schema

[
  {
    "name": "id",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "value",
    "mode": "NULLABLE",
    "type": "INT64"
  },
  {
    "name": "day",
    "mode": "REQUIRED",
    "type": "DATE"
  }
]

Assuming that you have a correct avro data file (an exercise for the reader) that contains records that include values in the day column that are the number of days since the epoch, you can run the following bq load command to load that data into your table.

 bq --project_id my_project load --source_format=AVRO --time_partitioning_type=DAY --time_partitioning_field=day --use_avro_logical_types my_dataset.my_table gs://my_bucket/*.avro
© 2005 Ryan Chapin