Configuring VSCode to Use Build Tags in GoLang to Separate Integration and Unit Test Code

Posted on by rchapin

One of the ways to separate unit and integration test code in golang is to use build tags.

A TLDR; on build tags is that it is a way to tell the compiler which source code to include or ignore when doing a build.

For example: if we have the following golang code:

// +build integration

package integration

import "testing"

func TestSomething(t *testing.T) {
	// Some test code
}

Then this code will only be compiled if we run the following command that includes the integration build tag.

go build -tags=integration

In order to separate out our unit test and integration test code we include the integration build tag in each file that is part of the integration tests, and the unit build tag in each file that is part of the unit tests. This is all well and good when we are writing code in a text editor and compiling and running our tests on the command line, but if we are going to use VSCode there are a few things that you need to do to configure the project so that the code is compiled correctly in the IDE while you are working on it and debugging your code.

  1. Click on the folder for your project in the Explorer
  2. Type CTRL+, to open up the Settings Editor
  3. Click on Folder below the search bar in the Settings Editor and select the folder for the project that you are configuring
  4. Click on Extensions > Go in the left-hand navigation of the Settings Editor and scroll down to Build Flags
  5. Click the Add Item button and enter the following: -tags=unit,integration
  6. Scroll down to the Test Tags section and click on Edit in settings.json then add the following to your json file so that it looks like the following
{
    "go.buildFlags": [
        "-tags=unit,integration"
    ],
    "go.testTags": "unit,integration",
}

You can also, just create a settings.json file in the .vscode directory in your project and add the aforementioned items to it. VSCode is really nice because it gives you a number of ways to manage your configs.

With these configuration changes VSCode will now be able to compile all of your application code, unit test, and integration test code while you are working on it.

VSCode Change Indent for File Explorer Tree

Posted on by rchapin

For me, I have a hard time distinguishing between the folders, sub-folders, and files in the file explorer of VSCode because the indent is not very pronounced.

In order to increase the indent go to File -> Preferences -> Settings. Search for “indent” and then click on Workbench in the left-hand side navigation pane of the settings window and scroll down to Workbench > Tree: Indent and then enter the number of pixels for the tree indentation. I doubled it, going from 8 to 16 and that made it much easier to see.

Visual Studio Code Cheat Sheet under Linux

Posted on by rchapin

As I’m still learning vscode, I’m keeping track of a cheat sheet for shortcuts and other functionality for the IDE.

The following is under Linux, specifically Debian 10 with XFCE.

Keyboard Shortcuts

  • Edit a string in the entire file that matches exactly the string that you have highlighted. For example: you might have a comment, or a variable name, in a file that you want to find and change globally. Highlight the string and then type CTRL+Shift+L then type the replacement string and it will replace it everywhere in the file.
  • Show All Opened Editors in Quick Open (edt command): CTRL+P
  • List all functions and symbols in a file: CTRL+P and then type @

VSCode Keyboard Shortcut to Navigate Between Split Panes

Posted on by rchapin

In general, I cannot stand using a mouse. It hurts my hand and takes up a ton of time having to move my hand from the keyboard, to the mouse, move the mouse, click, and then move back to the keyboard.

I am a relatively new VSCode user and was pleased that it has a really good vim emulator. I almost always write code in a split screen so that I can either see two different files at the same time, or see two different sections of the same file. I wanted to figure out how I could add a keyboard shortcut so that I could switch between split screens and found the following. See Dan Andreasson’s answer. As I write this the OP has yet to mark a correct answer.

TLDR; add the following to the keybindings.json file by typing CTRL+Shift+P and then enter Open Keyboard Shortcuts (JSON)

Under Linux this file should be in ~/.config/Code/User/keybindings.json

[
    {
        "key": "ctrl+h",
        "command": "workbench.action.navigateLeft"
    },
    {
        "key": "ctrl+l",
        "command": "workbench.action.navigateRight"
    },
    {
        "key": "ctrl+k",
        "command": "workbench.action.navigateUp"
    },
    {
        "key": "ctrl+j",
        "command": "workbench.action.navigateDown"
    }
]

[SOLVED] Deleting remote git branch; By default, deleting the current branch is denied, because the next remote: ‘git clone’ won’t result in any file checked out, causing confusion

Posted on by rchapin

This is a common error encountered where you are renaming your default branch for the repository. If you have a repo hosted on GitHub or some other third-party service, there is likely some way in the GUI to change the default branch for a repo.

If you are hosting your own internal git repository you will need to SSH to that server and “checkout” a different branch from the one that you are trying to delete. In reality, the remote repository has a branch “checked out”. To change it, ssh to the remote host and cd into the <project>.git dir and run the following command to see the current branch

git symbolic-ref HEAD

This will show you the current branch that is “checked out”. To change it run

git symbolic-ref <new-branch>

Once you have done that, you can delete the old default branch.

[SOLVED] Upgrading Apache Kafka 2.7 to Java 11 Changes authenticationID sent to ZooKeeper Enabling Only 1 Kafka Broker to r/w znodes

Posted on by rchapin

The title of this post is a bit of mouthful and requires a bit more explanation.

I am running a pure open-source version of Kafka (currently running 2.7) and am using SASL/GSSAPI connections between all of the brokers and ZooKeeper. Currently, the whole system, including ZooKeeper, is running Java 8 and it is long-overdue to be upgraded to Java 11.

Upgrading Kafka to Java 11 causes the server to send an incorrect authenticationID String to ZooKeeper which results in the ACLs on the znodes being set to the hostname of the first Kafka server that connects to ZooKeeper. This results in only one of the Kafka hosts being able to r/w the znodes.

Prior to upgrading the following are the logs from one of the ZooKeeper nodes:

2021-05-17 14:47:35,345 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.19.65.22:41698
2021-05-17 14:47:35,357 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.19.65.22:41698
2021-05-17 14:47:35,367 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x1797ac46a0f0000 with negotiated timeout 18000 for client /172.19.65.22:41698
2021-05-17 14:47:35,470 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET;  authorizationID=kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET.
2021-05-17 14:47:35,474 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@134] - Setting authorizedID: kafka
2021-05-17 14:47:35,474 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@964] - adding SASL authorization for authorizationID: kafka

The following is the corresponding logs from the Kafka host making the connection:

[2021-05-17 14:47:37,512] INFO Successfully authenticated client: authenticationID=kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET; authorizationID=kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET. (org.apache.kafka.common.secur
ity.authenticator.SaslServerCallbackHandler)

The most important part of this is that ZooKeeper sees the authenticationID as kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET. ZooKeeper is configured with the following options so that the host and realm will be removed from the authenticationID. This will leave the service name, ‘kafka‘, and result in all of the Kakfa znodes with kafka as the owner of the node. This then enables all of the hosts in the Kafka cluster r/w access to those znodes. Following are the configurations in ZooKeeper that enable this behavior.

kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true

The resulting znodes ACLs are

[zk: localhost:2181(CONNECTED) 5] getAcl /brokers
'world,'anyone
: r
'sasl,'kafka
: cdrwa

System Configurations

Following are all of the configurations for both the ZooKeeper and Kafka cluster:

ZooKeeper

zoo.cfg

maxClientCnxns=50
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/data_dir
dataLogDir=/zk/data_log_dir
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
clientPort=2181
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true

server.1=zk2-01.dv.quasar.nadops.net:2888:3888
server.2=zk2-02.dv.quasar.nadops.net:2888:3888
server.3=zk2-03.dv.quasar.nadops.net:2888:3888

jaas.conf

Server {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/zookeeper/conf/vmwqsrdvz201.dv.quasar.nadops.net.keytab"
  storeKey=true
  useTicketCache=false
  principal="zookeeper/vmwqsrdvz201.dv.quasar.nadops.net@EXAMPLE.NET";
};

Kafka

broker-jaas.conf

KafkaServer {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/usr/local/kafka/config/vmwqsrdvk01.dv.quasar.nadops.net.keytab"
  storeKey=true
  principal="kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET";
};

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/usr/local/kafka/config/vmwqsrdvk01.dv.quasar.nadops.net.keytab"
  storeKey=true
  serviceName="zookeeper"
  principal="kafka/vmwqsrdvk01.dv.quasar.nadops.net@EXAMPLE.NET";
};

server.properties

Only the relevant configs are included below.

listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,BROKER_LISTENER:SASL_SSL

inter.broker.listener.name=BROKER_LISTENER
listeners=SASL_SSL://:9092,BROKER_LISTENER://:9093

ssl.keystore.location=/usr/local/kafka/config/kafka01.dv.quasar.nadops.net-keystore.jks
ssl.keystore.password=***
ssl.key.password=***
ssl.truststore.location=/usr/local/kafka/config/kafka01.dv.quasar.nadops.net-truststore.jks
ssl.truststore.password=***
sasl.kerberos.service.name=kafka
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI
zookeeper.set.acl=true
super.users=User:kafka
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=false

Upgrading Kafka to Java 11

Once Kafka (leaving ZooKeeper on Java 8) is upgraded to Java 11 Kafka no longer sends the expected authorizationID String to ZooKeeper.

Following are the ZooKeeper logs of the same authentication:

2021-05-18 14:31:01,627 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.19.65.23:53374
2021-05-18 14:31:01,631 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.19.65.23:53374
2021-05-18 14:31:01,636 [myid:3] - INFO  [CommitProcessor:3:ZooKeeperServer@617] - Established session 0x3797fd27e300004 with negotiated timeout 18000 for client /172.19.65.23:53374
2021-05-18 14:31:01,728 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=vmwqsrdvk02@EXAMPLE.NET;  authorizationID=kafka/vmwqsrdvk02.dv.quasar.nadops.net@EXAMPLE.NET.
2021-05-18 14:31:01,728 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@134] - Setting authorizedID: vmwqsrdvk02
2021-05-18 14:31:01,728 [myid:3] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@964] - adding SASL authorization for authorizationID: vmwqsrdvk02

Following are the logs from Kafka that results in an exception being thrown because it is unable to write to the ZooKeeper znode:

[2021-05-18 14:31:01,410] INFO Initiating client connection, connectString=vmwqsrdvz201.dv.quasar.nadops.net:2181,vmwqsrdvz202.dv.quasar.nadops.net:2181,vmwqsrdvz203.dv.quasar.nadops.net:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@39b43d60 (org.apache.zookeeper.ZooKeeper)
[2021-05-18 14:31:01,414] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-05-18 14:31:01,419] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2021-05-18 14:31:01,421] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-18 14:31:01,556] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,573] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,577] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-05-18 14:31:01,604] INFO TGT valid starting at:        Tue May 18 14:31:01 UTC 2021 (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,604] INFO TGT expires:                  Wed May 19 00:31:01 UTC 2021 (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,605] INFO TGT refresh sleeping until: Tue May 18 22:32:58 UTC 2021 (org.apache.zookeeper.Login)
[2021-05-18 14:31:01,606] INFO Opening socket connection to server vmwqsrdvz203.dv.quasar.nadops.net/172.19.65.21:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-05-18 14:31:01,614] INFO Socket connection established, initiating session, client: /172.19.65.23:53374, server: vmwqsrdvz203.dv.quasar.nadops.net/172.19.65.21:2181 (org.apache.zookeeper.ClientCnxn)
[2021-05-18 14:31:01,622] INFO Session establishment complete on server vmwqsrdvz203.dv.quasar.nadops.net/172.19.65.21:2181, sessionid = 0x3797fd27e300004, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-05-18 14:31:01,626] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-18 14:31:01,731] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /brokers/ids
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:120)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:564)
        at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1662)
        at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1560)
        at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1(KafkaZkClient.scala:1552)
        at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1$adapted(KafkaZkClient.scala:1552)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.zk.KafkaZkClient.createTopLevelPaths(KafkaZkClient.scala:1552)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:467)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:82)
        at kafka.Kafka.main(Kafka.scala)
[2021-05-18 14:31:01,733] INFO shutting down (kafka.server.KafkaServer)

The authorizationID now provided by Kafka is now vmwqsrdvk02@EXAMPLE.NET, only the hostname and realm, and no longer the String that ZooKeeper expects. The resulting, derived authorizationID is vmwqsrdvk02 which means that only that Kafka host has r/w to any of the znodes which it creates.

Solution

After exhausting search options and finding multiple pages indicating that I have everything configured correctly, even for JDK 11, I posted to the #apache-kafka channel on chat.freenode.net and got a hint. Evidently, there are some JDKs that have bugs in their Kerberos implementation.

I was running java-11-openjdk-11.0.8.10-1.el7.x86_64 from CentOS 7 Updates. I double-checked and there was an updated JDK available.

After updating to the latest Java 11 JDK everything worked as expected. If the particular JDK that you are using is still not working try AdoptOpenJDK.

Crock Pot Black Bean Soup

Posted on by rchapin

Prep: 20 minutes; Cook: 2 hours; Total: 2 hours 20 minutes; 8 Servings

Ingredients

  • 3 15oz. cans black beans drained and rinsed
  • 1 4 oz. can green chilis
  • 1 cup chunky salsa
  • 1 cup vegetable broth
  • 1/2 medium white onion, finely minced
  • 3 tablespoons minced garlic
  • 2 tablespoons cumin
  • 1.5 tablespoons chili powder
  • 1 teaspoon paprika
  • 1/4 teaspoon salt
  • 1/4 teaspoon pepper

Instructions

  1. Place everything in a slow cooker, mix well, and cover.
  2. Cook on high for 2 hours, stirring halfway through.
  3. Uncover the slow cooker and mix well.
  4. Then, remove 3 cups of the black bean soup and place it into a high speed blender. Blend until the soup is completely smooth.
  5. Pour the black bean puree back into the slow cooker and mix well. There should be some back bean chunks left in the soup.
  6. Top with roasted or sauteed vegetables and enjoy!

Thai Curry Tofu

Posted on by rchapin

Ingredients

  • 1 tablespoon canola oil
  • 1 (12 ounce) package extra-firm tofu, drained and cubed
  • 1 tablespoon seasoned salt, or to taste
  • 1 tablespoon butter or margarine
  • 1 small onion, chopped
  • 3 cloves garlic, minced
  • 1 (10 ounce) can coconut milk
  • 2 teaspoons curry powder
  • ½ teaspoon salt
  • ¼ teaspoon ground black pepper
  • ¼ cup chopped fresh cilantro

Instructions

  1. See the following for instructions on how to cook the tofu.
  2. Melt butter or margarine in the same skillet over medium heat. Add the onion and garlic; cook and stir until tender. Stir in coconut milk, curry powder, salt, pepper and cilantro. Return the tofu to the skillet. Simmer over low heat for 15 minutes, stirring occasionally.

Baseline Settings for a VirtualBox Instance for a GUI under Debian 10

Posted on by rchapin

The following is a list of the basic VirtualBox settings to start with when using a VM with a GUI. Make sure that you have also installed the Oracle VM VirtualBox Extension Pack that is compatible with your version of VirtualBox and that you have installed the Guest Additions in the VM itself.

  • System
    • Motherboard
      • 4+GB of RAM, or whatever you have available
    • Processor
      • 2+ CPU (if you have the available cores/threads)
  • Display
    • As much Video Memory as you can spare
    • Graphics Controller: VMSVGA
      • Ensure that 3D Acceleration is enabled

© 2005 Ryan Chapin