-
Ansible Passwords
Ansible Vault lets us encrypt values against an external password. The resulting ciphertext can be included directly in the inventory and avoids the need to encrypt the entire file. If the password is kept externally from git so it is impossible to recover from the git repository alone. Lets set...
-
Confluent JSON Schema - Blog series
If youve ever wondered how Confluent JSON Schemas are managed and evolved there is quite a lot of reading and experimenting involved. To make things easier to understand Ive written a series of blog posts explaining how it all works. Since each post builds on the previous one, this index...
-
Which Confluent Schema ID?
kafkacat -u -b localhost:9092 -t azure -J| python3 -c "$(echo 'aW1wb3J0IHN5cwppbXBvcnQganNvbgpmb3IgbSBpbiBzeXMuc3RkaW46CiAgICBqID0ganNvbi5sb2FkcyhtKQogICAgcmkgPSBqWydwYXlsb2FkJ11bMTo1XS5lbmNvZGUoJ3V0Zi04JykKICAgIHNpID0gaW50LmZyb21fYnl0ZXMocmksICJiaWciKQogICAgcHJpbnQoZiJwYXJ0aXRpb246IHtqWydwYXJ0aXRpb24nXX0gb2Zmc2V0OiB7alsnb2Zmc2V0J119IHNjaGVtYSBpZDoge3NpfSIpCgo=' | base64 -d)" Why? Sometimes you just want to see the message partition, offset and schema IDs as messages are produced so that you can can concisely present information. Other approaches are the Confluent Control Center and the kafka*console-consumer*...
-
How to register JSON schemas in Confluent Schema Registry using the REST API
Create the schema By hand By generation (online, from POJO etc) Save the schema somewhere Example: schemas/value.json { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "properties": { "messageCreateTime": { "type": "string" }, "messageType": { "type": "string" }, "sourceApplication": { "type": "string" }, "targetApplication": { "type": "string" }, "tranID": { "type": "string" } },...
-
Confluent schema evolution in development and production
Schema Evolution: The official word TLDR Free-for-all when solo developing Practice your schema evolution when working as part of a team Defined process for schema evolution in production Schema evolution In development In a development environment you have a lot of flexibility in terms of what to do around schema...
-
Confluent JSON Schema Evolution
TLDR Take control of your JSON schema generation in production contexts If your just adding fields no need to worry just add them If field deletion is required, an open content model must be used In-depth Confluent JSON Schema evolution behaves differently to “normal” schema evolution. For the nuances, take...
-
Kubernetes + NVIDIA on K3S
Goal: Setup a Kubernetes node to expose NVIDIA GPU so that GPU loads (AI, Crypto, etc…) can run on Kubernetes: Platform: Debian 12 AMD64/x86_64 NVIDIA RTX 3070 Kubernetes (K3S) What are we trying to do? Expose a “GPU” capability on K8S nodes that have GPUs How do we do it?...
-
Problems with ancient date-time values
Sometimes you get questions that really lead you down rabbit holes: Year Zero My database uses 0000-12-31 as a timestamp and it crashes Google Big Query as it doesnt consider 0000 as a year. So to solve this customer asks if we can replace all 0000-12-31 values with 0001-01-01. Why...
-
Creating a new TypeScript project in 2023
Theres a few steps involved to make a new TypeScript project in 2023. There are lots of guides out there and their all “missing bits”: ES2022, testing, linting, etc - you have to assemble the pieces yourself. Workstation setup nvm Editor: WebStorm Visual Studio Code Project setup To avoid writing...
-
Adding typescript-eslint to a project
Heres how to quickly add typescript-eslint to your typescript project. Of course, this isnt the same thing as configuring eslint for typscript 😂 Step 0 Project already configured with typescript Step 1 Install typescript-eslint: # eg npm install --save-dev @typescript-eslint/parser @typescript-eslint/eslint-plugin eslint typescript Step 2 Use sensible defaults: Disable parseInt,...
-
macOS on AWS
Sometimes your mac colleagues need some instructions on how to do things or test interesting workarounds to Docker bugs MacOS does not run well in VirtualBox and this probably breaks licensing too - so how to test short of borrowing a computer? AWS - heres how!
-
Docker and K3d on macOS with bridged networking
Want to test Kubernetes but dont have a spare PC and want to save money on your cloud bill? Run kubernetes locally with Docker and K3d. Normally theres no bridge networking so you have to port-forward to get data in and out of the cluster. These instructions let you reach...
-
Confluent S3 Sink Connector EOS
Confluents S3 Sink Connector is an easy way to use Kafka Connect to dump data in AWS S3 object store. A useful feature of the connector is its ability to support Exactly Once Semantics (EOS) S3 and EOS EOS is notoriously hard to get right with S3 because S3 objects...
-
Switching to swapfile on linux
Many Linux installs use a swap partition instead of a swap file. There are some good reasons for this such as enabling sequential access for HDDs and hibernate support but a physical partition means if sizes are wrong a live USB and complicated resize procedure will be needed. These days...
-
Shrinking LUKS encrypted LVM2 EXT4 filesystem
Sometimes you need to shrink EXT4 filesystem hosted on LVM + LUKS to free up a bit of space for an additional filesystem. This can be vital in some situations: /boot too small and blocks kernel upgrades… but almost the whole drive is taken by LUKS - I give myself...
-
Nvidia drivers on Debian and Ubuntu
Official Documentation: Debian: https://wiki.debian.org/NvidiaGraphicsDrivers Ubuntu: https://help.ubuntu.com/community/NvidiaDriversInstallation Dont forget to ON the GPU in UEFI/BIOS! Debian enable non-free and contrib for all repos. Eg, update every entry in /etc/apt/sources.list: # old deb http://deb.debian.org/debian bookworm main non-free-firmware # change to deb http://deb.debian.org/debian bookworm main non-free-firmware non-free contrib apt install nvidia-driver firmware-misc-nonfree firmware-misc-nonfree...
-
Growing EXT4 LVM filesystem to use all free space
Growing filesystems is thankfully quite easy and can be done online. Most of the time I just want to use all available free space in a PV, to reclaim space freed by deleting another LV. There are just 3 steps: Take a backup Extend the LV to use all free...
-
Flatpak Firefox + import Firefox profile from Snap
Firefox on Debian Theres two “happy paths” with Firefox on Debian listed on the WIKI: Firefox ESR (Extended Support Release) - older/more stable version. Usually installed by default Flatpak/flathub latest Firefox If your coming from a recent Ubuntu you will need to use a recent Firefox since you cant import...
-
Fix/Setup grub + LUKS (fix grub/bootloader)
Did you just accidentally trash grub or manually install Debian using debootstrap? If so, this is how to get your system booting again by getting it to the stage where grub can be setup so you can rescue the system/finish the install. Boot environment to setup grub (adjust as needed)...
-
Fix Druva backup crash on Linux with Nvidia
If your running Druva InSync on a Linux box with Nvidia hardware, you might experience a crash every time you try to load the app. This manifests as the app starting and immediately exiting. The cause for this is incompatibility between Electron GPU acceleration and the Nvidia driver. Since this...
-
Upgrading to Debian 12 from whole disk Ubuntu + LUKS
Debian 12 is out and its awesome. My personal highlights on a 2017 era Thinkpad: Bundled firmware blobs Working out-of-the-box WIFI “Easy” NVIDIA driver setup Secure boot support 100% free (as in freedom) software… apart from those nasty blobs above Sound, bluetooth, webcam - 100% working No SNAPs If you...
-
Your own Kubernetes environment
How to build a complete Kubernetes environment on a single spare PC or EC2 VM, complete with storage, load balancer(s) and ingress. This is great for testing. For a more production grade setup add some more PCs or VMs. For easy DNS support, add static hostnames on your router for...
-
PostgreSQL Docker Environment
Here’s the quickest way to spin up and connect to a PostgreSQL Docker instance. podman is used as a drop-in replacement for Docker. Create a PostgreSQL server podman run --rm -p 5432:5432 --name postgres -e POSTGRES_PASSWORD=secret postgres:14.4-bullseye Connecting to the server PGPASSWORD=secret psql --host localhost --port 5432 --user postgres
-
MongoDB Docker Environment
Here’s the quickest way to spin up and connect to a MongoDB Docker instance. podman is used as a drop-in replacement for Docker. Create a MongoDB server podman run --rm -p 27017:27017 --name mongo -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=secret mongo:5.0.9 Connecting to the server mongo --username root --password secret
-
MySQL Docker Environment
Sometimes you just want to spin up a MySQL server without having to re-read the instructions. podman is used as a drop-in replacement for Docker in this example. You’re welcome. Create a MySQL server podman run --rm -p 3306:3306 --name mysql -e MYSQL_ROOT_PASSWORD=secret -d mysql:8.0.29-debian Connecting to the server mysql...
-
Tracing Maven transitive dependencies
Sometimes you might get a stray security alert for a maven project, eg my own atlassian plugin: https://bitbucket.org/declarativesystems/puppet_deploy_bitbucket/ was recently flagged on the Atlassian Marketplace for cve-2022-22965. This turned out to be due to a transitive dependency on spring-beans which we can prove by testing a couple of scenarios. The...
-
Headless Raspberry Pi setup
To setup a headless Raspberry Pi with WIFI and SSH access on Raspberry Pi OS follow these steps: Clone or fork https://github.com/declarativesystems/raspberrypi_configs Set a password for pi user Edit userconf.txt: pi:ENCRYPTED_PASSWORD ENCRYPTED_PASSWORD from echo 'mypassword' | openssl passwd -6 -stdin Configure WIFI: Edit wpa_supplicant.conf and change the placeholders to the...
-
Github Actions example
Introduction Lets take Github Actions for a spin by trying it out on a project. I have an existing repository to build container images for various crypto daemons so lets see how we go building it with Actions Review Github Actions is great. The documentation was spot on and everything...
-
K8s External Secrets
Developed by godaddy and given back to the community, External Secrets fill the gap between K8s secrets and secure credential storage. Externals secrets are defined as K8s resources like this: apiVersion: 'kubernetes-client.io/v1' kind: ExternalSecret metadata: name: dockerconfigjson namespace: someorg spec: backendType: secretsManager template: type: kubernetes.io/dockerconfigjson dataFrom: - /someorg/dockerconfig_secret In this...
-
Adding features to K8s
Not as hard as it sounds. K8s has an event stream and/or pageable lists of what the cluster is doing. A pod in a namespace with needed permissions should be able to ask or poll the K8s endpoint to see what needs to be done, eg registering external DNS or...
-
What is a Kubernetes ingress controller
Here’s the deal with K8s - it defines standards and protocols for a whole bunch of stuff but doesn’t necessarily provide an implementation. This is the case with ingress controllers. The K8s spec describes a bunch of HTTP routing behaviours with things like path based routing and default backends which...
-
what is Kubernetes
Kubernetes or K8s for short is the next big thing. Everybody’s doing it but what is it? And why would you want to use it? At its heart K8s is a container scheduling engine. Someone allocates compute and storage capacity to it and the system runs (Docker compatible) containers where...
-
what is upsert
UPSERT does exactly what it says it does: If a record is missing it INSERTs it Otherwise it UPDATEs it This means keeping data current is idempotent - less code is needed so there are less opportunities to make mistakes.
-
What is a database
From Wikipedia: In computing, a database is an organized collection of data stored and accessed electronically from a computer system. Where databases are more complex they are often developed using formal design and modeling techniques. Or to put it another way, it’s a bunch of data, probably stored on a...
-
Exposing version in JavaScript
Sometimes you want to expose the version number from package.json to your app so you can write code like this: console.log"Express server listening on port %d in %s mode %s", app.address().port, app.settings.env, app.VERSION This isn’t my code - there’s a whole discussion about how to export the version without having...
-
Cost debugging JFrog Pipelines
Sometimes you get really high CI costs that are unexplained: This graph is showing us we have consistently high downloads of about 30GB a day for a CI pipeline that isn’t actively running. This is data from JFrog cloud so we can use Artifactory Query Language (AQL) via Artifactory REST...
-
What is a homedir
A homedir or home directory is a place for you to store your own files: On a Mac its usually /Users On Linux its usually /home On Windows its usually C:\Users Your System Administrator may have changed this location. If your not human, you’re probably a Daemon (system) user. Daemons...
-
3D Printing Photographs
Sometimes you just want to turn a photograph into a 3D printed shape so you can share it with a friend: Cropping this leaves a nice greyscale image. We can think of the lighter areas as being high and the darker areas as being low. dem2stl is a script originally...
-
What Is A Bug
Legend has it that bugs were real insects that shorted the electronics of the original computers. Nowadays bugs are bad behaviour of computer software. It would be nice if these bugs could all be fixed but this isn’t usually achieved for large software projects. There are a few broad categories...
-
Recovering access to EKS
Sometimes things go wrong and you need to recover access to an EKS cluster using only the root account. Here’s one way of restoring the kubectl command to its former glory. According to stack overflow: if you don’t have authority to assume the IAM entity (user or role) that creates...
-
3D Printed toilet paper?
Declarative Systems specialise in infrastructure and business solutions but also experiment with developing technologies. One of those being 3D printing. Over in the Czech Republic, Prusa Research are doing some great things with this technology. If you have design skills and/or access to a lot of materials you should seriously...
-
Speeding up Ubuntu Docker builds with podman
Hacking around with Docker images on a laptop is a productive way to do experimental development but if your building Ubuntu images locally and have a lot of packages to apt install your going to be stuck waiting for downloads if you need to change something within a single layer....
-
No Python on RHEL/CentOS 8
! TLDR: Python is not pre-installed on RHEL/Centos 8, there is a secret Python 3 ! at /usr/libexec/platform-python but we are not supposed to use it. End users ! should install their own python. Beergeek 1:48 PM No python on RHEL8 is interesting Geoff 1:49 PM woah excuse me… RHEL...
-
Puppetize Live 2018 - Do More with Less
We’re very grateful to Nigel Kirsten and Puppet for the chance to speak at this year’s Puppetize Live event in Sydney. The presentation is a fun, and informative talk about rethinking how you write and deliver infrastructure code as a whole, not just specifically Puppet. You can check the presentation...
-
PDQTest 2.0 release
PDQTest 2.0 is finally out! PDQTest was created back in January 2017 and aside from being used on Declarative Systems modules and at few select clients, no one else seems to have heard of it. PDQTest is 100% free software (Apache 2.0 license) and came about because the current crop...
-
Integrating PDK
PDK has now well and truly arrived on the Puppet landscape. With Puppet 6, its mandatory in order to build Puppet modules (unless you stay on an older version of Puppet of course…). That means third party tools like PDQTest need to integrate PDK somehow or be stuck forever on...
-
PE_Kit - Demo Puppet Enterprise without an engineer
Do you need to sell a complicated product to customers and have to take Sales Engineer out on trips to do basic demos to Sales Qualified Leads? Before Declarative Systems existed, there was PE_Kit, a GUI for demoing Puppet Enterprise in seconds, not minutes. PE_Kit still works and is 100%...
-
Deploying a Private Puppet Forge
If your using Puppet or Puppet Enterprise with a Control Repository (which you definitely should be), then you may have noticed that the instructions mention two main ways of getting this working: Use the Puppet Forge directly Mirror modules to a local git server and use from there. Many larger...
-
Giving your Puppet Code the Onceover
Ever wanted to do some basic testing of your Puppet Control Repository but no idea how to get started? Dylan Ratcliffe from Puppet had this problem a couple of years ago and his answer was Onceover. What does Onceover do? Onceover lets you quickly and easily test your Puppet Control...
-
Puppet Custom Facts
Facter is an open-source tool for inspecting your system and is an integral part of Puppet. Facter gives users the ability to gather inventory information on a hundred or so different items on nodes connected to Puppet. Once Puppet knows about a particular fact, it will appear in the console...