James McNee - Blog

Fascinating Faceting with Postgres

James McNee — Sun, 05 May 2024 00:00:00 +0000

A facet is a piece of information about an entity that can be used to categorise it. Take for example a piece of fruit, it might have facets relating to its colour, size, weight, etc. Facets are a way of describing an entity in a structured way and are often used in search engines to allow users to filter results.

Let's continue with the fruit theme, imagine that you are a fruit vendor and you have created a website to sell your fruit. Customers love the quality of your produce, but they are often overwhelmed by the sheer variety of fruit that you offer. You think of other websites that sell products and how they allow users to filter their results by various attributes, such as price, colour, etc. You decide to implement this feature on your website too, and you want customers to be able to drill down to the type of fruit they wish to consume easily, you also want your users to see at each stage how many fruits will match the criteria they are applying.

You're a wiz with UI, so you squirrel away and create all the UX elements you need, this also helps you to visualise the data your server will need to respond with.

As you can hopefully see from the image above, the user can filter the fruit by colour, size and origin. When a user selects a colour, the other categories update to show the number of fruits that match the criteria they have selected, the numbers in the category to which the filter has been applied do not change, this shows the user how many additional results will be added if they select one of these. It's obvious that your customers will celebrate that they no longer have to scroll through pages of fruit to find the perfect apple.

Each facet can have multiple values, e.g. the colour facet could have values of red, green, yellow, etc.
A filter applied to a facet should only affect the counts of the other facets and not the facet to which the filter pertains. E.g. if a user selects red, the counts of the other colours should not change.
In a system with pagination, the counts should be calculated for the entire dataset, not just the current page. This means any global filters/search should be applied to the counts.

Faceting in Postgres

Let's have a look at a mechanism for implementing faceting in Postgres.

It's important to note here that this is intended to be implemented with the aid of a high level language such as Java, TypeScript, etc. and not just in SQL. This is because the query will need to be dynamic based on the user's selections.

The mechanism described below is not the most optimal way to implement faceting in Postgres, it should work great for moderate datasets, but for large datasets you may want to seek out a more performant solution.

Let's look at a sample of our dataset, we have a table called fruit:

Name	Colour	Size	Origin
Apple	red	medium	domestic
Banana	yellow	large	american
Blueberry	blue	small	domestic
Cherry	red	small	european
Durian	green	medium	american

It has columns for the name, colour, size and origin of the fruit. From this list of columns, we wish to facet on the latter three.

The first step for creating the facets is simply counting each of the values for each of the columns. We can do this with a query like the following:

SELECT facet_name, jsonb_object_agg(COALESCE(facet_value, 'null'), count) AS facet_values
FROM (
        SELECT facet_name, facet_value, COUNT(*) AS count
        FROM "fruit",
        LATERAL (VALUES ('colour', "colour"), ('size', "size"), ('origin', "origin")) facets(facet_name, facet_value)
        GROUP BY facet_name, facet_value
     ) facets
GROUP BY facet_name;

This yields an output of:

facet_name	facet_values
colour	`{ "red": 2, "yellow": 1, "green": 1, "blue": 1 }`
size	`{ "medium": 2, "large": 1, "small": 2 }`
origin	`{ "domestic": 2, "american": 2, "european": 1 }`

You can of course choose a different type of list aggregation other than `jsonb_object_agg` such as `string_agg` or `array_agg`. JSON is usually easier to parse in a high level language such as Java or TypeScript.

What we have above is great, it provides the counts for each of the facetable columns. However, we need to be able to filter these counts based on the user's selections. Let's change the query as if a user has selected colour: red and size: medium:

SELECT facet_name, jsonb_object_agg(COALESCE(facet_value, 'null'), count) AS facet_values
FROM (
        SELECT facet_name, facet_value, COUNT(*) AS count
        FROM "fruit",
        LATERAL (VALUES ('colour', "colour"), ('size', "size"), ('origin', "origin")) facets(facet_name, facet_value)
        WHERE "colour" = "red" AND "size" = "medium"
        GROUP BY facet_name, facet_value
        UNION ALL 
            SELECT 'colour' AS facet_name, "colour" AS facet_value, COUNT(*) AS count
            FROM "fruit"
            WHERE "size" = "medium"
            GROUP BY facet_name, facet_value 
        UNION ALL 
            SELECT 'size' AS facet_name, "size" AS facet_value, COUNT(*) AS count
            FROM "fruit"
            WHERE "colour" = "red"
        GROUP BY facet_name, facet_value
    ) facets
GROUP BY facet_name;

This yields an output of:

facet_name	facet_values
colour	`{ "red": 1, "green": 1 }`
size	`{ "medium": 1, "small": 1 }`
origin	`{ "domestic": 1 }`

Notice that we can still see colours and sizes that were not selected, but that have matching fruit from other applied filters. I.e. even though red was selected, there is still another green fruit that is also medium in size.

The query needs to be dynamically created based on the user's selections with various pieces of it being added and altered such as the where clauses. This is fairly trivial to do in a high level language but remember to ensure that user input is sanitised to prevent SQL injection attacks.

Conclusion

In this post, we explored the concept of faceting and how it can be implemented in Postgres. We also looked at how to filter the facets based on user selections. This is a great way to provide a flexible and powerful search mechanism to your users.

Managing permissions with Postgres event triggers

James McNee — Tue, 13 Feb 2024 00:00:00 +0000

Setting the scene

We (Auto Trader UK, where I work in the infrastructure team) use GKE (Google Kubernetes Engine) to run our application workloads. We use Google's CloudSQL offering to provide developers with relational data stores, such as MySQL and Postgres for their applications to integrate with.

Our in-house abstraction layer atop tools such as Helm, Kubernetes and Isito provides developers with a self-service interface to get an application spun up and deployed to production in minutes without the need to ask for servers to be provisioned, DNS configured and firewalls poked. Equally, our developers don't need to learn all the nuances of the aforementioned cloud technologies and can focus on delivering value in their language of choice.

Applications running within our clusters use Workload Identity to participate in Identity Access and Management (IAM), this grants the application access to Cloud Storage Buckets and CloudSQL without credentials, meaning no secrets, key rotation and overall improves security. In regards to Postgres, this means that a login role (a role which has the login attribute set) is bound to a Google Service Account, this provides authentication and we can then use Postgres' standard authorisation/permissions model for the rest.

Provisioning and management

We make heavy use of CRDs (Custom Resource Definitions) to model processes and infrastructure that is not natively provided by k8s core (or one of the third parties we integrate with, such as Istio) and we recently extended this to include the full provisioning and management of Postgres as a platform feature. Our users can now, from within their application repositories describe:

The Postgres instance they want, the major version it should be, its size, in terms of CPU and RAM and any flags that need enabling.
Their database name and any custom roles that should be defined along with the access these roles should ensure (more on this later).
Users and applications that should have access to the database besides the current workload and what role they should assume.

We strongly discourage applications sharing a datastore as a general rule, doing so (generally) creates tight coupling and leads to unexpected interaction, however, we have workloads such as Kafka Connect that fundamentally require it. Some older (pre-cloud) applications were written in a way that requires this access.

When our custom Kubernetes controller sees that one of these resources has been created, it will act to apply the desired configuration, be this provision a new instance, perform an in-place upgrade or create and configure a database.

We have an opinionated way in which our databases are configured and we allow configuration from users only within the bounds we have set, for example, our controller will create an 'owner' role that the application assumes when it connects, this role has CREATE permissions on the database and is expected to be the owner of all objects within, no other role is granted this access. It is the responsibility of the application to create relations (tables, views) etc either in code or by using a technology such as Flyway.

We also opt to never grant permissions directly to login roles themselves, but instead create higher order roles that we dub 'permissions roles', due to them being granted permissions. We then set the ROLE configuration parameter of the login role to this permissions roles name, meaning that on login they will assume that role. This gives us the flexibility of managing users separately from the database (in terms of K8s resources) and also allows us to configure the permissions of groups of users by altering a single role.

Custom roles

As briefly mentioned, we allow developers to define custom roles that should be created on their database. These roles can then be shared with other workloads (such as Kafka Connectors, Data Importers etc) to allow them restricted access to the database.

A custom role definition that a developer may define for a Kafka Connector, might look like this:

postgres:
  databases:
    myDatabase:
      name: my_awesome_database
      instance: some_instance
      roles:
        kafka-connect:
          permissions:
            perSchema:
              financial_reporting:
                specific:
                  tables:
                    outbox:
                      - SELECT
        sharedWith:
          'my-kafka-connector':
            roles:
              - 'kafka-connect'

The above may seem a little verbose, with which I would tend to agree, however, it provides application developers with the power to configure as broad or fine granularity as they desire. In the above example, a role has been defined called kafka-connect and it has been given SELECT permission on a single table named outbox in the financial_reporting schema. This role will not be able to access any other database objects or modify the data in the one table it can access.

Not only does providing this level of granularity protect from unexpected actions by the user of the role, either intentional or not, it also provides for clear auditing (in both git and the K8s cluster) of what access has been extended to other workloads. In the above example we can see the role kafka-connect has been shared with the workload named my-kafka-connector (workloads are one-to-one with namespaces in our clusters and are therefore unique).

Two problems to overcome

Having such a flexible schema for expressing permissions brings two main challenges that need to be overcome

If a role is described that grants to 'all' objects within a schema, how do we ensure the role accrues permissions to new objects created after the granting code has been run?
If a role is described that grants to a specific object, how do we account for the fact that it may not yet exist?

Let's explore these two issues...

Perpetual permissions

When a developer defines their role and deploys it via their CI/CD pipeline as part of their application release, it is fairly trivial for our code to apply the grants to 'every object' that exists in the specified schema, assuming they have defined a role that should do this. The harder part is keeping this statement true as new objects are created after the fact.

Postgres has a system called default privileges which essentially solves the issue above, it allows for describing the permissions that should be granted when new objects are created at either a database or schema level. In terms of this requirement, default privilege is a valid option.

An ordering problem

Described in the previous section is a mechanism for defining and sharing custom roles with other workloads, you will note that the example role gave specific access to a single object (table) in a single schema. This is all well and good, assuming that the subject of the grant already exists, if not we would not be able to set up our permissions role.

This posed more of an issue for us than might be immediately obvious, as it seems only logical that users would create tables etc before adding them to a custom role. Ultimately, yes it is logical for this to happen and as a series of events, very may well happen, but one needs to remember the way we are managing the datastore, using a K8s CRD ultimately deployed via helm. This means that as soon as an application's CI/CD pipeline runs it (helm) will create both the applications deployment and the resource representing the database, these will and must be able to handle being applied in any order.

If a developer has a bunch of commits stacked up such as (ordered as a git log, top = latest):

da06bf8 Grant access to the orders table to kafka-connect
c7662cd Implement API for handling orders and persisting to the database
0f9c839 Add SQL migration script to create new orders table

It's possible and most likely that the database resource will be applied and handled before the application rolls out and creates the table, meaning that when the code creating the role tried to run GRANT SELECT ON orders TO some_role; it would have received an error.

There are a few ways to potentially tackle this, some of which might be:

Document it as a 'known gotcha' and hope that developers will learn to roll out changes in isolation, e.g. add the script to create the table, roll it to prod, then add the grant.
Make the controller essentially swallow the error and ignore it if the object doesn't exist.
Teach Postgres how to dynamically assign the permissions.

Spoiler alert, it's the third one that we went with and to which this article pertains! But let's explore those other two.

Document it as a 'known gotcha' and hope that developers will learn to roll out changes in isolation, e.g. add the script to create the table, roll it to prod, and then add the grant.

Adding documentation for quirks and calling it a day is a last resort option really, at best it's annoying for users to have to remember this gotcha and realistically people will understandably forget.

Make the controller essentially swallow the error and ignore it if the object doesn't exist.

There are a few issues with this one, but the main one is that unless the spec of the resource changes the controller will not run. Therefore, if the above scenario plays out as described, the grant will not be applied and then will also not try again until another unrelated change to the resource spec is made.

Event Triggers to the rescue

So how can you grant permissions on an object that doesn't (yet) exist? Well, as far as I know, you cannot. But all hope is not lost, because what if we can somehow hook into the lifecycle (create, update, delete) of objects in the database and run our grants at this point? Luckily Postgres has a mechanism for this, Event Triggers.

What is an event trigger?

Triggers have long existed on relational databases as a way to hook into DML (Data Manipulation Language) actions such as INSERT and DELETE on rows in tables for example, however, standard triggers are not able to observe DDL (Data Definition Language) events such as CREATE and DROP. Postgres' Event Trigger mechanism though is built to facilitate this and allows for the execution of code at either the beginning or end of one of these commands. A function that hooks into these events can observe and even reject the action, making them a powerful tool for restricting actions.

The most common use case for event triggers seems to be for centralised auditing of DDL tracking when, and by whom was a table, view, sequence, etc) created, dropped or modified. Another use case discussed online in various places is around restricting the types of objects that can be created, for example blocking the creation of functions, materialised views etc.

The event trigger function, whether it is invoked at the start or end of the DDL command, is fully involved in the transaction. This means that if an exception is raised at any point, the whole transaction will be aborted and rolled back if required.

Hopefully, it's becoming clear how we can leverage this to achieve the functionality we desire, granting permissions across the board not only on objects that currently exist but also on ones that do not yet. To quote a popular maxim, it allows us to "kill two birds with one stone" (only metaphorical birds were harmed, don't worry!).

Defining the event trigger and function

The functions and procedures that are described in this section will be generated and applied on the fly when the k8s resource is observed. For a single database or manual management, this may be quite hard to maintain and simply be overkill.

Without further ado, let's have a look at an event trigger based, on the configuration example above:

First, we will define a procedure (a function that doesn't return a value) that will contain all our granting logic, it will take the following arguments:

object_id: The oid of the object that is currently being handled.
object_identity: The identity (schema-qualified name) of the object.
object_type: An enumerable mapping for the type of the object is RELATION (Table, View), SEQUENCE etc.
schema_name: The name of the schema to which the object belongs.

We will then template our function with the logic that checks if the current object is one we are interested in and if so, applies the grants.

-- Function to inspect the current object and apply grants upon it if desired.
CREATE OR REPLACE PROCEDURE admin_schema.handle_grants(object_id oid, object_identity text, object_type text, schema_name text)
 LANGUAGE plpgsql
 SET search_path TO 'admin_schema', 'pg_temp'
AS $procedure$
    BEGIN
      IF object_type = 'SCHEMA'
      AND EXISTS (SELECT nspname
                  FROM pg_catalog.pg_namespace
                  WHERE nspname = object_identity
                  AND nspowner::regrole::text = 'owner_role')
      THEN
        EXECUTE format('GRANT USAGE ON SCHEMA %s TO "kafka-connect";', object_identity);
      END IF;
      
      
      IF object_type = 'TABLE' object_identity = 'financial_reporting.outbox' THEN
        EXECUTE format('GRANT SELECT ON %s TO "connect";', object_identity);
      END IF;
    END;
    $procedure$

The next thing we need is a function that the event trigger will invoke directly, the shape of this function is rigid in that it must return an event_trigger type and also has access to retrieve data about the current invocation of the trigger. The above function could be folded into this, however, to foreshadow slightly, it will be useful to have it separate later...

There are a few interesting things about this function:

It uses SECURITY DEFINER - When a function is invoked in SQL, usually it will be done so acting as the CURRENT_USER i.e. the current role that is set for the session. However, it's often necessary to use a function to allow a user to perform elevated actions they could not otherwise perform, without giving them too much power. By creating a function as SECURITY DEFINER it means that the function should be executed as the role that owns it, rather than the invoker's role.
It calls a mystery pg_event_trigger_ddl_commands() function and loops through the returned rows. This function returns the 'DDL commands' that caused the event trigger to fire, usually, this will only contain a single row, but certain statements can result in multiple.

Hopefully the rest is fairly self-explanatory, we loop through all the DDL commands that have happened in the current invocation and call our grant procedure after grouping the DDL commands by their target. I.e. tables, views and materialized views should all be treated as RELATIONs.

-- Function to be invoked directly by the event trigger
CREATE OR REPLACE FUNCTION admin_schema.handle_role_grants_event_trigger()
 RETURNS event_trigger
 LANGUAGE plpgsql
 SECURITY DEFINER
 SET search_path TO 'admin_schema', 'pg_temp'
AS $function$
    DECLARE
        obj RECORD;
    BEGIN
        FOR obj IN SELECT * FROM pg_event_trigger_ddl_commands()
            LOOP
                IF obj.command_tag IN ('CREATE TABLE', 'CREATE VIEW', 'CREATE MATERIALIZED VIEW', 'CREATE TABLE AS') THEN
                    CALL "admin_schema".handle_grants(obj.objid, obj.object_identity, 'RELATION', obj.schema_name);
                end if;

                IF obj.command_tag = 'CREATE SEQUENCE' THEN
                    CALL "admin_schema".handle_grants(obj.objid, obj.object_identity, 'SEQUENCE', obj.schema_name);
                END IF;

                IF obj.command_tag = 'CREATE SCHEMA' THEN
                    CALL "admin_schema".handle_grants(obj.objid, obj.object_identity, 'SCHEMA', obj.object_identity);
                END IF;
            END LOOP;
    END;
    $function$

And finally for all of this to work, we need to define the event trigger and bind it to our function, which thankfully is simple!

Notice in the following snippet that we specify ddl_command_end, there are two main points one can hook into:

ddl_command_start - Execute (and wait for) the function BEFORE running the DDL itself, inspecting catalogues at this point will not show the change as being reflected. This is a good point to perform validation/security checks.
ddl_command_end - Execute (and wait for before committing) AFTER running the DDL itself, inspecting catalogues at this point will show the change. This is what we want as we need the object to exist to apply grants upon it.

As mentioned earlier, both of these happen within the transactional boundary of the DDL event, so if the function throws (raises an exception) then the whole transaction will be rolled back.

CREATE EVENT TRIGGER role_grants_event_trigger ON ddl_command_end
    EXECUTE FUNCTION "admin_schema".handle_role_grants_event_trigger();

Reconciliation

The above section describes how we can set up an event trigger to respond to new objects being created on the database and dynamically grant the appropriate permissions to roles that should have access, however, it's important to also be able to 'reconcile' existing objects on the database and have permissions applied retrospectively.

A few use cases where this is needed:

The first time the event trigger is added to an existing database.
If a new object is defined as requiring roles to have access that previously did not.
Each time the access list is altered either allowing more roles access to an object or changing the permissions an existing one has.
In the event that something unexpected happened and the grants were not applied.

Essentially, it's going to be important to run a full reconcile each time the spec for the Kubernetes resource changes, this catches all of the above and also will allow for applying changes to the granting logic and having it retrospectively applied.

Rather than maintaining two completely disparate procedures, one for responding to DDL events and one for reconciling existing objects, we can utilise the slightly abstracted procedure (admin_schema.handle_grants) that we created above. This ensures that the granting logic is the same no matter which code path triggers it.

This is the final function/procedure, I promise! Here's what it's doing:

Applying grants

Finding all objects by querying pg_class
For each object, invoke (call) the handle_grants procedure, exactly the same as the event trigger does

Revoking old grants

Finding all privileges on objects that are in schemas that are owned by the owner_role (described in a section above).
Loop over each of these privileges (grants)
If the grant is still valid, move on, and leave it alone.
If the grant is no longer valid, run a REVOKE.

CREATE OR REPLACE PROCEDURE admin_schema.reconcile_role_grants()
 LANGUAGE plpgsql
 SET search_path TO 'admin_schema', 'pg_temp'
AS $procedure$
    DECLARE
        row RECORD;
    BEGIN
        -- Logic for reconciling role grants
        FOR row IN
            SELECT oid,
                   concat(quote_ident(relnamespace::regnamespace::text),
                          '.',
                          quote_ident(relname)
                   ) AS object_identifier,
                   relnamespace::regnamespace::text AS schema,
                   relkind
            FROM pg_catalog.pg_class
        LOOP
            -- relkind - r: Table (Relation), v: View, m: Materialized View
            IF row.relkind IN ('r', 'v', 'm') THEN
              CALL "admin_schema".handle_grants(row.oid, row.object_identifier, 'TABLE', row.schema);
            END IF;

            -- relkind - S: Sequence
            IF row.relkind = 'S' THEN
              CALL "admin_schema".handle_grants(row.oid, row.object_identifier, 'SEQUENCE', row.schema);
            END IF;
        END LOOP;

        FOR row in (SELECT nspname AS schema FROM pg_catalog.pg_namespace)
        LOOP
            CALL "admin_schema".handle_grants(NULL, row.schema, 'SCHEMA', row.schema);
        END LOOP;

        -- Revoke grants that are no longer valid
        FOR row IN
          SELECT *
          FROM (
            SELECT relnamespace::regnamespace::text AS schema,
                   relname AS object_name,
                   (CASE
                      WHEN relkind = 'r' THEN 'TABLE'
                      WHEN relkind = 'v' THEN 'TABLE'
                      WHEN relkind = 'm' THEN 'TABLE'
                      WHEN relkind = 'S' THEN 'SEQUENCE'
                    END) AS object_type,
                   (aclexplode(relacl)).grantee::regrole::text AS grantee,
                   (aclexplode(relacl)).privilege_type::text AS privilege
            FROM pg_class
            WHERE relacl is NOT NULL AND relkind IN ('r', 'v', 'm', 'S')
          ) privileges
          WHERE EXISTS (SELECT nspname
                        FROM pg_catalog.pg_namespace
                        WHERE nspname = schema
                        AND nspowner::regrole::text = 'owner_role')
        LOOP

          IF row.object_type = 'TABLE' AND row.schema = 'financial_reporting' AND row.object_name = 'outbox' AND row.grantee = 'kafka-connect' AND row.privilege = 'SELECT' THEN
            CONTINUE;
          END IF;

          EXECUTE format('REVOKE %s ON %s FROM %s;',
                          row.privilege,
                          concat(quote_ident(row.schema), '.', quote_ident(row.object_name)),
                          quote_ident(row.grantee));
        END LOOP;
    END;
    $procedure$

Conclusion

The functions and procedures that have been shown in this post are very much an MVP of what gets applied to the databases that are being managed by this mechanism, but hopefully, this has shown how it all fits together and provided a base for anyone wishing to do similar.

As mentioned earlier in the post, we have an opinionated way that our databases should broadly look and therefore can do things like scope to things owned by the 'owner role', not everyone will be in this position.

It'd be interesting to hear any thoughts on what we are doing here and how others have solved similar issues, especially when trying to fully automate database provisioning and management.

Waiting is hard... but just Persevere(JS)

James McNee — Sun, 15 Oct 2023 00:00:00 +0000

Preface

Working with technologies that are asynchronous in nature, such as messaging queues like Apache Kafka and RabbitMQ, brings not only some new interesting mechanisms for communication but also a few challenges to deal with in terms of verifying they are working as expected.

I'm a big fan of what I like to call 'shallow integration tests', these are tests that cover the integration around a single slice of your application, without testing the whole thing (a full integration test). Let's explore this a little further as it'll help with context.

Shallow Integration Testing

In order to properly explain what I mean by 'shallow integration tests', it's somewhat important to touch on those either side of them, namely unit and integration tests.

Unit Tests are those that are focused on the testing of a single unit of code, e.g. a function/method. They are isolated in what they test, meaning that they do not rely on external factors such as the network or a database. The aim of unit testing is to prove that given a set of inputs/conditions the isolated unit responds/behaves as expected.
Integration Tests differ from unit tests in that rather than testing an isolated piece of code such as a function or method, they test the integration between all the units of your application. Integration tests tend to be higher level than a unit test, not exercising every edge case, but rather testing core flows through your application.

When I refer to a 'shallow integration test' what I am referring to is a test that sits at a boundary of your application and tests the integration between your code that interacts with an external system (or is interacted with). Most commonly this will either be at the repository or controller layer.

A repository is a class that is responsible for handling data exchange between your application and an external system, this could be a database, HTTP service, message queue or even something like a printer.
A controller is a class that handles the interaction into your application, this could be a set of HTTP endpoints, a console/terminal prompt a consumer of a messaging queue or even a physical button that someone might press.

Imagine that we have an application that serves a HTTP API for a library, one of the functions that this API performs is allowing patrons to check if books are available to borrow. The API is backed by a MongoDB database which it can use to track and update the availability of books.

Preventing Catastrophe using Gum

James McNee — Sat, 07 Oct 2023 00:00:00 +0000

It's all about context

There are a variety of binaries that I use daily that utilise a static 'context' so that the commands run against a specified environment. These tools have some state, most commonly a file on disk somewhere that references the context they are currently set to.

It's too easy when using tools like kubectl, helm and gcloud to inadvertently run a command against the wrong cluster or project because you have forgotten that you have switched to production. Hopefully, the command is ultimately harmless, but it could easily not be.

This post is to share how I help to mitigate the risk of accidentally running destructive commands against production infrastructure using gum, which is an awesome tool that allows developers to build powerful TUI (terminal user interface) based scripts.

Of course, an argument can be made around not having the level of privilege to do anything dangerous in production contexts in the first place, but it's like all things a trade-off between security and practicality. Having access to production clusters is necessary at some level to be able to respond to incidents and investigate issues, how this access is managed is not the topic of this post, but is in itself an interesting topic.

Visual signals

Before I get onto the meat of the post, around how I use gum to help prevent accidents, it's good to note some of the other techniques that both myself and others use to aid with this goal, because as with most things in life, more is often better.

Custom Prompts

Most shells provide a mechanism for customising the 'prompt' i.e. where user input is taken. Customisations range from showing the current status of a project in version control, how long a command took to execute, the exit code of a command and so on.

A very useful customisation that one can add to their shell is to show the current Kubernetes cluster that they are pointing at and if desired, the namespace they have active too. You can see this in practice in the above screenshot.

Colour Coded Shells

Another approach, although not one that I use, is to have the title bar of the terminal window change colour depending on the current context. Many people choose to have their shells getting more and more red as they step through environments closer to production.

Gumming it up

The above two approaches are good and do help to prevent mishaps, but they still rely on the user noticing a visual cue. What I wanted was something that would pull the emergency break for me and prevent me from slamming into the wall that is, a production incident.

What is Gum?

Gum is a binary that can be used to build powerful TUI (terminal UI) based scripts quickly and easily, check out the readme on their GitHub for some examples and how to use it.

Essentially though, gum allows you to add dynamic user input into an otherwise boring shell script, notable features are:

Input requires the user to enter some input, this can be used inline or assigned to a variable.
Write allows the user to provide a block of multi-line text.
Filter and Choose will show a list of choices, the former allowing the user to type to filter. Supports multi-selection etc.
Confirm is what we will be focusing on. This shows a prompt to the user seeking confirmation.

Creating a kubectl wrapper

I am going to focus on kubectl for the remainder of this post, but this technique can be applied to basically any binary. I use it for both kubectl and helm currently.

The first thing I needed to do was to create a wrapper script that I would invoke whenever I typed kubectl (or my alias k). I could then capture the command and perform the sniff test to see if it looks dubious. You can see what the result of this looks like in the screenshot above, just so you know what we are building.

I have a pattern for creating my shell scripts and an automated way of sharing dependencies between them, which I may write about at some point, but for now, I will keep it simple. I tend to have a main function to contain any logic, this is mainly for variable scoping, but it also aids with reading. First, I create two variables, one that will point at the real kubectl binary and another that will get the currently active kube context.

#!/bin/bash

function main() {
  local KUBECTL_PATH="${HOME}/kubernetes/kubectl"
  local CURRENT_CLUSTER=$(${KUBECTL_PATH} config current-context)
}

main "$@"

Now, let's take an iterative approach. We will get something working quickly and then enhance it, so firstly let's just only allow actions to take place if the context is `testing'.

#!/bin/bash

function main() {
  local KUBECTL_PATH="${HOME}/kubernetes/kubectl"
  local CURRENT_CLUSTER=$(${KUBECTL_PATH} config current-context)
  
  if [[ $CURRENT_CLUSTER != "testing" && -n $1 ]]; then
    exit 1
  fi
  
  "${KUBECTL_PATH}" "$@"
}

main "$@"

With the above, we are aborting if the current kube context is not testing and if it is, we continue, passing through the original arguments to the real kubectl binary.

Now, rather than just exiting early, let's make the script ask us to confirm if we intend to perform the action.

#!/bin/bash

function main() {
  local KUBECTL_PATH="${HOME}/kubernetes/kubectl"
  local CURRENT_CLUSTER=$(${KUBECTL_PATH} config current-context)
  
  if [[ $CURRENT_CLUSTER != "testing" && -n $1 ]]; then
     if ! gum confirm "WARNING: This will run against '${CURRENT_CLUSTER}'. Are you sure you want to do this?" --default="false" --timeout="10s"; then
        echo "CANCELLED: Aborting command; that was close!" >&2
        exit 1
      fi
  fi
  
  "${KUBECTL_PATH}" "$@"
}

main "$@"

Awesome, now, by using gum, we will be asked if we are sure that we wish to execute the command against a production cluster.

We could stop there, as this fulfils the objective that we set out with, but there are some operations that I don't need to be asked to confirm, even for production. Things like

kubectl get when looking at cluster state
kubectl logs when grabbing pod logs
kubectl version when looking at what binary version I am running

So let's enhance the script to exclude those, and some more 'safe' commands. What you deem to be a safe command may differ, and that is fine, edit as appropriate.

#!/bin/bash

function main() {
  local KUBECTL_PATH="${HOME}/kubernetes/kubectl"
  local CURRENT_CLUSTER=$(${KUBECTL_PATH} config current-context)

  local SAFE_OPERATIONS=(
    "get"
    "explain"
    "describe"
    "logs"
    "completion"
    "config"
    "api-resources"
    "version"
  )

  local SAFE_OPERATIONS_PIPE_DELIMITED=$(IFS="|"; echo "${SAFE_OPERATIONS[*]}")

  if [[ $CURRENT_CLUSTER != "testing" && -n $1 ]]; then
    if echo "$1" | grep -qvE "^(${SAFE_OPERATIONS_PIPE_DELIMITED})$"; then
      if ! gum confirm "WARNING: This will run against '${CURRENT_CLUSTER}'. Are you sure you want to do this?" --default="false" --timeout="10s"; then
        echo "CANCELLED: Aborting command; that was close!" >&2
        exit 1
      fi
    fi
  fi

  "${KUBECTL_PATH}" "$@"
}

main "$@"

Boom! Now we will only be asked to confirm potentially destructive actions (edit, delete, scale and apply etc).

You will notice that I have added `completion` to my list of safe commands, this is to allow for kubectl autocompletion to work still, which by the way it very much does with this mechanism. Snazzy.

Ensuring that the wrapper is invoked

Now, we have a script that wraps kubectl, but now we need to make sure that it actually gets invoked, not only when I type kubectl or use an alias, but also if any other application tries to, because I don't want that modifying production either!

To achieve this, I created another very small script and called it kubectl, which I then placed in a location which is referenced in my PATH. This script simply executes the shim we created above. This looks like the following

#!/bin/bash
/path/to/where/shim/is/located/kube-shim.sh "$@"

I then created an alias for my preferred k invocation in my .zshrc file that invokes this shim script. Now, anything that attempts to use kubectl will be routed through the shim script. You could also if you wanted, just use the one script (i.e. place the shim in a bin folder and call it kubectl) but for my setup, I separated them.

Summary

To summarise, having a safety net around 'dangerous' operations on production infrastructure is blissful, obviously, even with mechanisms in place to prevent accidents from happening, complacency should be avoided.

Hopefully, this post will help you create a similar safety net around some of your binaries, or at least introduce you to gum, which is awesome.

Hardening Security: From Zero to Hero

James McNee — Sun, 01 Oct 2023 00:00:00 +0000

Preface

I think it's important to preface this article with the following disclaimer... I am by no means a security expert! Everything that you read here is just documentation of what I have implemented for my blog at JamesMcNee.co.uk. As you will read though, I have not just made this up off the top of my head. I have used the tools that have been written by security experts to evaluate the hardiness of a website.

With this in mind, this post serves to explain how I took my blog (the one you're reading now, unless I have rebuilt it again!) from having a weak D- score over at observatory.mozilla.org to a smug A+ in a couple of hours (and cups of coffee ).

But what makes a site 'secure'?

Good question, and one that until I saw my abysmal score I didn't know either! Well, that's not quite true, there are obvious things like not sending passwords in plain text, using TLS/SSL over HTTPS rather than HTTP etc, but this post focuses on the more nuanced ways that a site can be exploited.

All of this stemmed from running a scan on observatory.mozilla.org after I saw it linked in a post I was reading. I had already been using Google's Lighthouse to improve the performance of my site, so when I hit run on the Mozilla Observatory, I braced myself to be informed on how brilliant I am (*cough* Eleventy and Cloudflare are). So when the results came in and I had been awarded a D- I could only laugh with shame.

So what was wrong?

Well, Observatory found the following issues with my site (I will be exploring these in more detail in the coming sections, so below are the descriptions paraphrased from MDN):

[-25 Points] No Content Security Policy (CSP): An added layer of security that helps to detect and mitigate certain types of attacks, including Cross-Site Scripting (XSS) and data injection attacks. These attacks are used for everything from data theft, to site defacement, to malware distribution.
[-20 Points] No HTTP Strict Transport Security (HSTS): Allows a website to inform the browser that it should never load the site using HTTP and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead. It consists of one HTTP header, Strict-Transport-Security, sent by the server with the resource.
[-20 Points] No X-Frame-Options (XFO): Used to indicate whether a browser should be allowed to render a page in a <frame>, <iframe>, <embed> or <object>. Sites can use this to avoid click-jacking attacks, by ensuring that their content is not embedded into other sites.
[-10 Points] No X-XSS-Protection (XXSS): A feature of Internet Explorer, Chrome and Safari that stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. These protections are largely unnecessary in modern browsers when sites implement a strong CSP.

If one thing is clear, it's that my team will not be winning whatever game we're playing with a scorecard like that... There were some positive results in the report too, alas I cannot take credit for those, they were because my host (Cloudflare Pages) had taken care of them for me.

Let's work through that list, adding a CSP seems to be the biggest hitter as it will not only wipe out that top negative report (CSP), but it will also remove the last one (XXSS).

Implementing a Content Security Policy (CSP)

I have an admission to make, although I have heard those three letters banded around both online and in my workplace, I never really understood what a CSP was. It always seemed to be some mythical beast that front-end folks would cheer about once conquered as if they had just slain a dragon. This gave me reason enough to avoid knowing more about it, that is of course until I did not really have a choice!

So, assuming that you have already read the definition above from MDN, and pulled a face at the vagueness of it, I shall attempt to summarise. A CSP is an HTTP header that instructs clients (browsers) where the server expects content to be loaded from, thus preventing assets from loading from an unknown source. It describes where things like images, video, scripts and styles are allowed to originate from and each of these are individually addressable within the header.

A brief note on the `<meta>` tag

It's possible to emulate HTTP response headers (to varying degrees of success) using the <meta> tag, and setting the http-equiv attribute to the name of the header you're trying to use. If you have never come across this concept before, you can read more about it over on this page by w3, the standards authority for the web.

It is possible to have a CSP implemented using a <meta> tag, but there are some caveats (beyond the fact that using this tag is discouraged), the main one is that CSP reporting (more on this later) is not supported using this approach. Some additional features that are also not supported when using the meta tag approach, you can read more about them over at content-security-policy.com.

As touched on, a CSP header comprises rules targeting the various pieces of dynamic content that a site can load, here are the ones that were important to me, along with where I wanted to allow content from:

img-src: Allow images to be loaded from any origin.
media-src: Allow audio/video to be loaded from any origin too.
style-src: Allow styles to only be loadable from my own origins jamesmcnee[.co.uk | .com], along with styles from giscus, which I use for my post comments section.
script-src: Allow scripts to be loadable from my own origins and giscus.
frame-src: Allow i-frames etc to connect to my own origins and giscus.
frame-ancestors: Do not allow my site to be i-framed, anywhere. If I need to do this for my own origin in the future, I'll open it up.
object-src: Do not allow <object> and <embed> elements.

Most of these rules seem simple enough, the tricky part comes from inline-scripts and inline-styles i.e. the <script> and <style> tags, and also directives like onclick="doSomething()" and style="color: blue;".

Why would inline scripts and styles be a problem?

I am going to focus on the first of these two because the exploit is easier to demonstrate, but you can read more about how styles can be exploited in this insightful and detailed Stack Overflow answer.

Imagine the following script that displays what the user searched for:

<script type="text/javascript">
    const searchTerm = getQueryParam("term");
    
    document.getElementById('search-results').innerHTML(`<h1>${searchTerm}</h1>`);
</script>

This looks harmless enough, but what about if I set my url to mydomain.com?term=<script>alert("EVIL")</script>? All of a sudden I have injected code into the webpage (alert is used as a display that arbitrary code can be executed), this could be exploited to make the browser do anything!

Obviously, if a bad actor wants to do this to their own browser, then all the power to them, the issue comes when this URL is hidden inside a link somewhere else and an unsuspecting party clicks it.

Okay, I am convinced, inline scripts are bad... But they are so convenient... I had a few small inline scripts on various pages throughout my blog that made sense to live there. This brings me to the workaround/solution for the inline script problem...

You can do one of four things, listed in order of general preference sentiment:

Move any inline scripts into separate script files and load them into your site as you would for external dependencies.
Use a nonce (number once) to bind the HTTP response from the server and the scripts returned in the body. As demonstrated here.
Take a hash of the script's content and add this to the CSP to mark the script as safe/allowed.
Last resort, you can allow inline-scripts in your CSP, but this greatly weakens the protections a CSP provides. Think carefully before doing this.

In my case, I used a combination of 1 (shifting scripts to separate files) and 3 (Using a hash of inline scripts/styles), I also removed any random style attributes that I had placed on elements with tailwind classes. I will not detail how to generate the hash, because it's explained well on the content-security-policy.com website, as are most CSP concepts, so it's worth a browse.

I actually believe that using the nonce technique is a good option, and if it was easily doable for my setup, I may have chosen to do this rather than move scripts out into their own files. Alas, because I have a static site, hosted by Cloudflare Pages, I cannot easily inject a nonce value onto each script tag per request without using something like CF workers. I'd rather avoid the additional complexity and potential cost (you get so much for free), and just use the other two mechanisms.

A brief note on CSP reporting

CSP's provide the ability to report violations to a given endpoint. I did not end up actually using this feature, but it can be a good way to implement a CSP in a limited fashion, by enabling report only mode, to make sure that nothing is missed.

You can read more about this feature, including how to utilise it, over on MDN here.

Building the CSP

Right, let's actually build the CSP header.

A CSP header looks like the following:

  Content-Security-Policy: default-src 'self' domain.tld *.domain.tld; img-src *; ...

Each directive (e.g. default-src, image-src, etc) is separated by a semi-colon (;) and each source (e.g. domain.tld) by a space.

My host (Cloudflare Pages) supports the somewhat well-known pattern of accepting a _headers file that defines additional response headers to serve on content, many static site hosts such as Netlify and GitHub pages also subscribe to this. This file takes the form of a path match pattern and then key-value based headers. e.g.

/blog/*
  X-MY-CUSTOM-HEADER: Value

In the above example, any request to a path that starts with /blog e.g. /blog/my-page would have the X-MY-CUSTOM-HEADER response header attached.

Initially, I began to implement the CSP header manually by adding to my _headers file a Content-Security-Policy header on the /* root. But this quickly became unwieldy to work with as I ended up needing to duplicate strings (like my domain for example). As I am using Eleventy (which I briefly wrote about my switch to in this previous post) I decided that the _headers could just be a templated file and I could then use Javascript to add a shortcode to build my CSP.

So now I had shifted the big unwieldy string from a text file into Javascript, which whilst better, was still pretty grim having to do lots of string interpolation. Before I went off and built a little function to generate the CSP, I googled and found that someone already had created a content-security-policy-builder library! Horray! This library allows you to pass in a structured object representing the CSP and it will do the string templating for you.

In the end, this is what I ended up with

eleventyConfig.addShortcode('csp', () => {
    const cspBuilder = require("content-security-policy-builder")

    const myDomains = ['jamesmcnee.com', 'jamesmcnee.co.uk']

    const defaultSrc = `'self' ${myDomains.join(' ')} ${myDomains.map(d => '*.' + d).join(' ')}`

    const allowedInlineScriptHashes = [
        'sha256-smKXypSFxzKD9ffC0rSshp292sAzf/X7cquCvQEA8XA=' // The post search script on index
    ]

    return cspBuilder({
        directives: {
            'default-src': defaultSrc,
            'img-src': '*',
            'media-src': '*',
            'style-src': `${defaultSrc} https://giscus.app`,
            'script-src': `${defaultSrc} ${allowedInlineScriptHashes.length > 0 ? `'unsafe-inline'` : ''} ${allowedInlineScriptHashes.map(hash => `'${hash}'`).join(' ')} https://giscus.app`,
            'script-src-attr': `'unsafe-hashes' 'sha256-1jAmyYXcRq6zFldLe/GCgIDJBiOONdXjTLgEFMDnDSM='`, // This is to allow the preloading of stylesheets
            'frame-src': `${defaultSrc} https://giscus.app`,
            'frame-ancestors': `'none'`,
            'object-src': `'none'`
        }
    })
})

Now, let me first address `unsafe-inline` here, before I get comments (I wish) telling me that I said it was bad and then went and used it! Well it's actually considered best practice to add `unsafe-inline` as a fallback for older browsers that don't support the hashing of scripts, if the browser does, it will be discounted. Therefore, I conditionally add it based on if I have a script hash or not!

I can then use this shortcode in my _headers template file which now looks like

---
permalink: _headers
---
/*
  Content-Security-Policy: {% csp %}

The {% csp %} here will be expanded during the eleventy build to the actual full CSP. In-case you're curious, this is what the templated file looks like using the above settings in the builder

/*
  Content-Security-Policy: default-src 'self' jamesmcnee.com jamesmcnee.co.uk *.jamesmcnee.com *.jamesmcnee.co.uk; img-src *; media-src *; style-src 'self' jamesmcnee.com jamesmcnee.co.uk *.jamesmcnee.com *.jamesmcnee.co.uk https://giscus.app; script-src 'self' jamesmcnee.com jamesmcnee.co.uk *.jamesmcnee.com *.jamesmcnee.co.uk 'unsafe-inline' 'sha256-smKXypSFxzKD9ffC0rSshp292sAzf/X7cquCvQEA8XA=' https://giscus.app; script-src-attr 'unsafe-hashes' 'sha256-1jAmyYXcRq6zFldLe/GCgIDJBiOONdXjTLgEFMDnDSM='; frame-src 'self' jamesmcnee.com jamesmcnee.co.uk *.jamesmcnee.com *.jamesmcnee.co.uk https://giscus.app; frame-ancestors 'none'; object-src 'none'

Right-o CSP done, +25 points, +10 points for handing XSS protection (implicit in the CSP) and +5 points for having a strong CSP! Next...

Implementing HTTP Strict Transport Security (HSTS)

Next on the agenda was to tackle HSTS, which if you remember is essentially not allowing HTTP traffic and enforcing that it be upgraded to HTTPS.

This one is much simpler than the last (you will be glad to hear), it is essentially just setting a response header, as I described at the end of the previous section, I can do this on my host using a _headers file. The header consists of three elements:

Max Age ^(*required): The time, in seconds, that the browser should remember that a site is only to be accessed using HTTPS. Recommended to be set as 2 years.
Include Sub Domains: If specified the rule will be applied to all subdomains as well as the current domain.
Preload: An optional parameter than when specified (must also have a max age of > 1 year, and include subdomains set to true) will include your domain in a list that browsers use to determine if your domain should only use HTTPS, without even making a request.

For this one, I just applied all of the settings, I set my max age as 2 years, included sub-domains and opted into the preload register. My _headers file now looks like:

---
permalink: _headers
---
/*
    Content-Security-Policy: {% csp %}
    Strict-Transport-Security: max-age=63072000; includeSubDomains; preload

Another one done, +20 points for ~~gryffindor~~ me! Next!

Implementing X-Frame-Options (XFO)

The final one on the list was to tackle XFO, you will remember that this is essentially setting whether your site should be allowed to be rendered inside an i-frame etc. Allowing your site to be embedded can be used to facilitate a click-jacking attack, whereby buttons on your site, could be hidden behind elements so that the user doesn't know that they are clicking them.

This header X-Frame-Options can be set to either DENY or SAMEORIGIN, the former blocks being embedded altogether, while the latter allows for you to i-frame your own site.

As I do not have a good reason to allow this, even on my own origin, I opted to set this to deny. My final _headers file looks like this

---
permalink: _headers
---
/*
    Content-Security-Policy: {% csp %}
    Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
    X-Frame-Options: DENY

Boom, that's a wrap, final one done, +20 points and an additional +5 points too apparently!

Summarising

If you made it this far, thanks, I hope it was useful! In this post, I covered how I took my static Eleventy site from having a security rating of D- score over at observatory.mozilla.org to A+ in a few hours, and now, hopefully, you can do the same!

The main things that needed doing on my site, and that I covered were:

Implementing a strong CSP to ensure that content is only being loaded from origins I expect
Adding an HSTS header to ensure that HTTPS is always used when accessing my site...
Adding an XFO header to ensure that my site is not i-framed anywhere, helping to reduce click-jacking

Adding dynamic search to a static Eleventy site

James McNee — Fri, 29 Sep 2023 00:00:00 +0000

Switching to Eleventy

Until recently, my blog at JamesMcNee.co.uk was a single-page application built using the Angular framework a few years ago. I decided to build the blog using Angular because it was, and still is, the framework that I am most comfortable with. In my day-to-day work, if I need to whip up a quick web app for users to interact with a system that I am building, my go-to will be an Angular-based SPA.

With time, I started to call into question my decision to build a blog using the Angular framework. The site was using an old version of the framework and upgrading was taking work, it was using a bespoke custom CMS that I built, powered by a Java API and MongoDB store. It also had a lot of custom CSS, which is not my forte. I wanted something lighter, faster to load and easy to host and maintain.

It was time to explore one of these new-fangled static site generators that everyone has been raving about, promising to solve all the problems I had with my current set up. After a bit of googling, I decided to try out Eleventy, and I was immediately impressed by both the minimalism and flexibility it provides. So a few hours later, I had ported the pages and content, mainly keeping in the same style as the previous incarnation, but with the extra goodness of using Tailwind to do it.

Searching for a way to search

The benefits that a static site provides in terms of web performance and the ease of adding new content are great, but it does pose a challenge for certain features that would be trivial with an API, in this case, search... I wanted to add a search component to my blog to allow for quick finding of posts by title and synopsis keywords.

So, like any good developer, I immediately went to Google in hopes of finding someone who has already done the work for me! Alas, my search did not yield a good example of how to implement what I wanted.

Building it from lab

After not having found a good example (there may be good examples, I just did not find one), I set in thinking about how I could implement it myself.

The back of the napkin requirements were simple, the solution should:

Plug into Eleventy's collection framework, so that I could use the same collection of blog posts for rendering and searching.
Have the ability to dynamically show results within a component, and not require linking off to a separate page.
Be entirely totally self-contained within the repository, i.e. not using something like Google Site Search.

Creating a search payload

The first step to implement this was to get a data source in place, a simple JSON page served on a static route should do it. Here is an example of a nunjucks template that gives the search information for each post on my blog:

search.json.njk

---json
{
    "permalink": "search.json",
    "eleventyExcludeFromCollections": true,
    "metadata": {
        "url": "https://www.jamesmcnee.com/"
    }
}
---
{
    "results": [
        {%- for post in collections.posts | reverse %}
        {%- set absolutePostUrl = post.url | absoluteUrl(metadata.url) %}
        {
            "url": "{{ absolutePostUrl }}",
            "path": "{{ post.url }}",
            "title": "{{ post.data.title }}",
            "synopsis": "{{ post.data.synopsis }}"
        }
        {% if not loop.last %},{% endif %}
        {%- endfor %}
    ]
}

This template yields an endpoint at /search.json with the following:

{
  "results": [
    ...
    {
      "url": "https://www.jamesmcnee.com/blog/posts/2023/sep/29/eleventy-search/",
      "path": "/blog/posts/2023/sep/29/eleventy-search/",
      "title": "Adding dynamic search to a static Eleventy site",
      "synopsis": "This post covers how I went about adding a dynamic search element to a static Eleventy based blog, without compromising on the benefits of SSG."
    },
    ...
  ]
}

This should be enough information to facilitate a search by keyword in the posts title or synopsis text and allow for linking off to the full article.

Creating the markup

Next, we need a search component, essentially a text input which can show search results below it.

    <div class="flex">
        <span class="flex-1"></span>
        <div>
            <div class="form-control w-full max-w-xs">
                <label class="label" for="search">
                    <span class="label-text">Search Posts:</span>
                    <span class="label-text-alt">e.g. "Patch"</span>
                </label>
                <input id="search" autocomplete="off" type="search" placeholder="Type here" class="input input-bordered w-full max-w-xs" />
            </div>
            <div id="search-results-container" class="card shadow-xl bg-base-100 absolute border-gray-600 border-solid border-2 translate-y-2 left-0 ml-[2%] w-[94%] z-50 md:left-auto md:w-96 md:-translate-x-1/2 lg:-translate-x-1/4 invisible">
                <div id="search-results" class="card-body p-4 text-sm max-h-96 overflow-y-scroll"></div>
            </div>
        </div>
    </div>

Search Posts: e.g. "Patch"

Post One

This is the synopsis for the first post, it gives a brief description as to its content.

Post Two

This is the synopsis for the second post, it gives a brief description as to its content.

The above is what I came up with, if you want to use it for inspiration, you can find the full markup for it over on my GitHub.

Implementing the dynamic results

Now that I had the markup for searching and displaying the results, I just needed to write a bit of Javascript to wire it up to the search payload we created earlier... For this, I just used an inline <script> block as it seemed the simplest solution.

<script type="text/javascript">
    function searchShouldBeVisible(visible) {
        document.getElementById('search-results-container').classList.remove(visible ? 'invisible' : 'visible')
        document.getElementById('search-results-container').classList.add(visible ? 'visible' : 'invisible')
    }

    function search(term) {
        // Set search results to invisible if the term is falsy (empty string, or null/undefined)
        if (!term) {
            searchShouldBeVisible(false)
            return
        }

        // Set search results to visible
        searchShouldBeVisible(true)

        // Fetch the full search payload
        const searchResponse = await fetch('/search.json')
        const responseBody = await searchResponse.json()

        // Filter the results array for items that contain the term (ignoring case)
        const filtered = responseBody.results.filter(item => `${item.title}${item.synopsis}`.toLowerCase().includes(term.toLowerCase()))

        // Get the DOM element to populate with results
        const resultsDiv = document.getElementById("search-results")

        // Special handling if nothing found
        if (filtered.length === 0) {
            resultsDiv.innerHTML = 'Nothing found...'
            return
        }

        // Build up the inner HTML for the search results div
        let compiledString = ''
        for (let i = 0; i < filtered.length; i++) {
            const post = filtered[i]
            const card = `<div class="cursor-pointer" onclick="window.location = '${post.path}';"><a class="font-bold mb-0" href="${post.path}">${post.title}</a><p class="mb-0">${post.synopsis}</p></div>`
            compiledString = `${compiledString}${card}`

            if (i !== filtered.length - 1) {
                compiledString = `${compiledString}<span class="divider mt-0.5 mb-0.5"></span>`
            }
        }

        // Assign the compiled string to the innerHTML for the div
        resultsDiv.innerHTML = compiledString
    })
</script>

The above is a slightly simplified version of what I ended up with as I also wanted the following features:

Click outside detection - When the user clicks or taps outside the search results, it should dismiss/hide them.
Search debounce - As I will be binding to the keyup event, to request the search payload for every user keystroke would be inefficient. Therefore, it would be beneficial to wait for the user to finish typing before running the search.

You can, if desired, have a gander at the full implementation of this on the GitHub repository for this blog. Do note that some of the required functions are off in other files though.

Tying it all together

All that is left now is to wire up the search input to the function we have created above.

<input id="search" autocomplete="off" type="search" placeholder="Type here" class="input input-bordered w-full max-w-xs" 
       onkeyup="search(event.target.value)" />

Summary

In this post we have explored how we can add a bit of dynamic flare to an otherwise static site and implement a useful and fully customisable search, whilst not having to leave the framework!

If desired, you could extend this to also:

Show a thumbnail for each search result
Dismiss the results when clicking outside (see above)
Debounce the search element (see above)

Gradle Plugin: Adding an external dependency

James McNee — Sat, 01 Jan 2022 00:00:00 +0000

I recently took on a bit of a side project to create a grade plugin to consolidate some shared build time concerns across a number of projects at my workplace.

While I thoroughly enjoyed writing the plugin and learned a lot about how Gradle works (I would recommend the exercise to all as a learning experience), I did hit some roadblocks along the way.

One of these problems is the focus of this post which should hopefully provide some help to others who are trying to solve the same problem.

Problem statement:
I would like my plugin to add an external dependency to the project it is applied to.

Jargon:

Let's just start by defining some terms that will be used that may not be obvious as to what they refer to.

Configuration: In Gradle, a configuration in the context of dependencies is the scope that it will be applied within. For example, in a Java project, the common scopes that we see are implementation and testImplementation. These two scopes define if the dependency will be available for use in the main application code or only in tests.

Diving into the code, let's imagine that we have a brand new plugin (this post will not cover how to create this, but official documentation can be found over on the Gradle docs site). Out main plugin class looks like this:

public class MyAwesomePlugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {

    }
}

Adding an external dependency is in theory not actually that much of an issue, we can simply use the project.getDependencies().add() method. The main issue comes when determining what to pass into this method. Let's take a look at the signature of the method: Dependency add(String configurationName, Object dependencyNotation). We know what configurationName is (see jargon section above), but what on earth are we supposed to pass as dependencyNotation?

String Notation: Simply a String written using Gradle dependency notation e.g. com.mycompany:my-awesome-dependency:1.2.3. There are also ways to specify things like strictness when using these 'simple' declarations.
Map Notation: This is where you pass a Map<String, String> containing key-value pairs representing the dependency. The documentation on this is either non existent or elusive, but for example: "group": "com.mycompany", "name": "my-awesome-dependency", "version": "1.2.3".

It is also possible that you pass in an object that implements one of the Dependency interfaces that the Gradle API provides. I would recommend against this though as there are no default implementations provided and you would therefore need to implement it yourself. This may not be such an issue but there are some methods on the interface that may require somewhat complicated logic to satisfy.

public class MyAwesomePlugin implements Plugin<Project> {

    @Override
    public void apply(Project project) {
        project.getDependencies().add("implementation", "com.mycompany:my-awesome-dependency:1.2.3");
    }
}

And that is it! It's not terribly complicated but it's also not immediately obvious, so hopefully, this post helps someone out.

Closing Remarks:

The Gradle API is very flexible and does a lot of 'magic', I have found myself fighting with this and trying to keep things nice and typed but sometimes you have just got to let Gradle do its thing!

Reverse Proxies: Take care what is being exposed!

James McNee — Sat, 20 Feb 2021 00:00:00 +0000

Having a microservice architecture certainly has advantages; it allows a service to be fully independent meaning that it can be both deployed and scaled separately while also allowing for a smaller code base which often helps developers to keep the codebase relatively clean. Microservices also allow for a trimmed down, domain-specific API which is not often the case when dealing with a monolith as it can be tempting to pull in multiple modules to build a single API response; this makes it hard to decouple and refactor.

One of the downsides to using this architecture though is that it moves away from the single URL / API' approach that is the preferred / most common way to interact with the services of a company. To illustrate this, imagine that a social network company wishes to allow developers to access some of its information, let's call this company... TwitBook. There are 3 data points that TwitBook wish to expose: User Data, Post Data and Message History. Each of these three data points are available internally via three separate apps, each with their own URL and REST API. TwitBook does not wish to expose these APIs as three individual, detached services but would like them to all appear as one with an API structure such as this: developer.twitbook.tld/api/user/{userId}, developer.twitbook.tld/api/post/{postId} and developer.twitbook.tld/api/message/{messageId}. Having endpoints exposed as a single API is also the most standard way for a front-end to communicate with its back-end service.

In order to have our cake and eat it too, have a detached microservice architecture but the benefits of a unified API, we can leverage a piece of technology known as the reverse proxy.

What is a reverse proxy?
A reverse proxy in its simplest form is a service that sits in front of multiple other services in which traffic (usually from the internet) is routed via. This layer can be used to add a layer of security, load balance, cache content and/or act as a facade for a microservice architecture (which this post focuses on).

Further reading around what a reverse proxy is and how it compares to a regular proxy can be found online such as

this article from Cloudflare.

Sorry... It's time for the other shoe to drop

So far, the reverse proxy sounds like a pretty ideal solution to creating the facade of a unified API within a microservice ecosystem, and it absolutely is. The problem that this post describes is not from the use of the technology itself but rather from misuse of it and a potential oversight when configuring a reverse proxy.

If we consider the earlier example of TwitBook and the desire to expose a single API that provides data from multiple services, a naive configuration for a reverse proxy might look something like this:

{
  "/api/user": "user-service.internal.twitbook.tld",
  "/api/post": "post-service.internal.twitbook.tld",
  "/api/message": "messaging-service.internal.twitbook.tld"
}

Here we are forwarding any request to developer.twitbook.tld/api/user to the internal user service and are doing similar for both post and message endpoints. At first glance, everything seems fine here and unfortunately, this is sometimes where the configuration ends. We are able to access any current endpoint and any endpoint that is added to one of these services in the future; hopefully, now the issue is starting to become clear.

The intention seldom is to expose every endpoint of the downstream service, this is especially true if the service in question has admin endpoints exposed. One could argue here that the admin endpoints should be exposed via a different port or at the very least have auth that is different from that of the 'regular' endpoints but not always!

I'm convinced! How can we solve this?

It's pretty simple really, only forward the endpoints that you actually want to expose and to go one step further... restrict the HTTP methods that are forwarded too. Here is a simplified example config illustrating this:

{
  "/api/user": {
    "target": "user-service.internal.twitbook.tld",
    "whitelist": [
      {
        "path": "/",
        "methods": ["GET", "POST"]
      },
      {
        "path": "/{id}",
        "methods": ["GET", "PUT", "PATCH", "DELETE"]
      }
    ]
  }
}

The implementation of this will depend on the framework that is being used for the reverse proxy, but hopefully, the idea has been conveyed. Using this method alone is not the only steps that should be taken to secure the proxy, it is vital to ensure that there are sufficient auth and user-level privileges in most cases too, but is a step up from just blindly forwarding everything.

To PATCH or not to PATCH; that is not the question

James McNee — Sat, 20 Feb 2021 00:00:00 +0000

When creating an API I will usually plan to implement 4 of the HTTP verbs; namely GET, POST, DELETE and PUT. These four request methods are all that are needed to implement a CRUD (Create, Read, Update and Delete) interface. It is also common to use the GET twice to implement a search interface, extending the acronym to SCRUD (or CRUDS).

There is another common HTTP verb that has not been mentioned above though and that happens to be the subject of this post; I of course refer to PATCH and not HEAD, CONNECT, OPTIONS or TRACE which are also not mentioned but are a little more on the obscure side.

Recently I decided to implement a PATCH method onto an API I was working on as it seemed appropriate. The context around this was to allow a boolean field to be updated and the original state was irrelevant to the client. One such use case would be notifications and marking them as read/acknowledged. With a newfound interest in PATCH I began researching the standards governing it and was surprised to find that there are in fact two main types of PATCH functionality which I will explore in this post; one being a Merge Patch and the other a JSON Patch.

Let's first assume no knowledge of the PATCH request method, and for that matter, we can also assume no knowledge of PUT. Both of these methods allow for a client to instruct the server to update a specific resource. The PUT verb uses a replacement method whereas the PATCH verb allows for more surgical alterations. Imagine a simple API that provides methods to interact with a user object and there is a user available at the following location: /api/user/1.

The current state of this user is as follows:

{
    "id": 1,
    "forename": "John",
    "surname": "Doe",
    "email": "john.doe@mail.com"
}

How a PUT request works

The PUT verb allows us to replace the entire state of the resource with what is being provided in the request body. For example, if we wanted to alter the email of the user above we could do the following request: PUT: /api/user/1.

{
    "forename": "John",
    "surname": "Doe",
    "email": "john.doe@newmail.com"
}

On the face of it, this has done exactly what we wanted, the result is that the email has been updated. The downside of this method is that we need to provide the entire state of the object again meaning not only that our request is bigger than it needs to be but also that we need to know the current state of the resource before updating it.

How a PATCH request works

The PATCH verb allows us to selectively replace a section of the state in a more surgical way than that provided by the PUT method instead allowing us to alter (or add) individual keys in the JSON document.

As mentioned in the preamble at the beginning of this post, there are two types of PATCH implementation. Let's take a look at both:

The 'Merge Patch' implementation

This implementation is perhaps the most common one and is the one that I have favoured when implementing a PATCH method. The merge patch works by only acting upon keys that are present in the request. An important thing to note here is that this implementation of the PATCH verb does not provide a mechanism to treat a null value separately than a remove operation i.e. if knowing that a value was explicitly set to null is important then this implementation will not allow for it. If we wanted to update the email of the user (just like we did in the PUT example) then we would execute the following request: PATCH: /api/user/1.

{
    "email": "john.doe@newmail.com"
}

In this case, we only provided the field that we wished to update; there was no need to provide the full state. The spec for the merge patch can be found: here (RFC 7396).

The 'JSON Patch' implementation

This implementation uses a format called JSON Patch which is a way to describe updates to a JSON document. This format defines a JSON schema that provides for six 'operations'; Add, Remove, Replace, Copy, Move and Test. If we wanted to use the JSON Patch format to perform our user update then we would execute the following request: PATCH: /api/user/1.

[
    {
        "op": "replace",
        "path": "email",
        "value": "john.doe@newmail.com"
    }
]

Again we did not need to provide the full state of the object and instead provided instructions on how to update the original document with our desired change.

Operations are passed inside of a JSON array allowing for multiple operations to be provided at once. There are a few benefits to this implementation of the PATCH verb. For one it is more explicit by providing the desired operation, it allows for differentiation between the value of null and the removal of a field if this is desired. The spec for the JSON patch can be found: here (RFC 6902).

How does the server know the difference...

... and what if I want to accept both types of patch?

The Content-Type header can be used to instruct the server which type of patch you are using. Providing application/json as the value can be used to denote a Merge Patch and providing application/json-patch+json should denote a JSON Patch.

Another header Accept-Patch can be used by the server to advertise which types of patch are supported by returning the values mentioned above. See more about this header here.

Summary

After putting in the time to understand and appreciate the PATCH request method I have come around to the thinking that having it is perhaps more useful than having a PUT endpoint in many ways. But as the title "To PATCH or not to PATCH; that is not the question" suggests, whether or not we decide to implement PATCH is not the real question. It is how we might go about it.

In this post, I explored two different methods of implementing a PATCH endpoint. I currently prefer the 'Merge Patch' method but do find the 'JSON Patch' implementation interesting and worth consideration.

Below are some resources that I used to help research the content in this post.

An interesting StackOverflow question and set of answers exploring PATCH endpoints in detail. The answers also explore the idempotency of the PATCH request method which is a very interesting read. This can be found here.
A great article on the differences between PUT and the two implementations of the PATCH endpoint. This can be found here.
RFC 6902 (JSON Patch). This can be found here
RFC 7396 (Merge Patch). This can be found here

Preventing log spam in Java

James McNee — Sat, 15 Aug 2020 00:00:00 +0000

Logging when something unexpected happens in an app is often a useful endeavour and provides valuable information to help diagnose and remedy potential errors. Logging infrastructure such as Elasticsearch (with a UI such as Kibana) allow for the easy storage and retrieval of log messages; ingesting and storing this data however has a cost.

In the everyday running of an application it might be normal to see a dozen messages in the logs. In this case it is very useful to have as much information as possible about each individual log to make sure that the diagnosis and remedy is as swift as possible. While it is useful to have this level of detail when there are so few logs, it is not very useful if there are thousands all pointing to the same error.

It is inevitable that an app will at some point run into an issue; this could be a simple issue such as loss of network connection. In this scenario it is important not to flood the logs with messages that could span into the thousands. Doing so puts strain on logging infrastructure but also makes it even more difficult to see other logs that may have happened around the same time.

This issue will be especially prevalent with asynchronous operations such as Kafka consumers where the number of messages is not always consistent.

In order to mitigate this issue I have created a CountingWindowBuffer class which once a defined threshold is hit will act as a circuit breaker to prevent the aforementioned log spam.

private final ThresholdBuffer buffer = new CountingWindowedBuffer(5, Duration.ofMinutes(1),
                (count) -> LOGGER.error(String.format("Something catastrophic has happened %d times... This is a disaster!!", count)));
                
public void handleSomeAsyncAction(Action a) {
  try {
    //...
  } catch (Exception e) {
    buffer.increment(() -> LOGGER.error("Some more specific text about this particular error + the original exception", e));
  }
}

Notice in the above example we defined a threshold of 5 and a window of 1 minute, this would mean that once the increment() method is called 5 times the buffer would activate and the supplied function will no longer be called. Instead once the window has elapsed, the function defined within the buffers constructor is executed. The number of executions within the window will also be provided to this function.

Using the above CountingWindowBuffer we can maintain the standard logging that we find useful while also ensuring that during an incident that may cause a drastic increase in logs we do not spam our logs.

You can find the code for this buffer over on my GitHub here.

Propelled by a Tailwind

James McNee — Sun, 03 May 2020 00:00:00 +0000

I participate in both front-end and back-end development; focusing mainly on the latter. I do however do a fair amount of front-end development, mostly in Angular, both professionally and for hobby projects. One of the main components of front-end development is the style of the site, defined in CSS (Cascading Style Sheets). It is fair to say that I do not excel at design and CSS; I can get by for hobby projects, and understand how CSS works, I simply do not have the passion for it. I believe that the single pain point and that which takes a huge chunk of time when creating personal projects is writing CSS. Each time I start a project, I think 'this time I shall do write the CSS in a maintainable way', yet each time it ends up in this delicate balance where it seems one style change could send the whole site out of whack.

With this thinking in place; that when starting a new project I need to start again with building this delicate CSS Jenga tower, it was a great surprise when listening to an episode of the Fish and Scripts podcast one of the hosts mentioned a library called Tailwind. This library was unlike many of the other CSS frameworks out there such as Bootstrap, which aims to provide ready-made components and classes that can be plugged onto elements such as a <div> to instantly render a predefined utility like a date picker. Tailwind, on the other hand, is different, it provides classes that add a single CSS property such that m-4 which adds margin: 1rem, the power comes from combining many of these classes to quickly build a component, without worrying about class names, or targeting the right element.

Tailwind helps you to use the DRY (don't repeat yourself) principal while also helping to prevent this from happening too early. When writing CSS, you will often find yourself writing in a way that means that you can reuse the class, but often it will only be used once, on that specific element. Tailwind supports creating custom classes that combine multiple tailwind classes, so you can create classes such as .btn and .link where reuse is required.

Anyway, that is enough advertising for the framework. I have only used it once in a limited fashion, and I am not sure how easy/nice it would be to integrate with an existing project. I am however really excited about the potential this could provide for me to abstract the CSS creation on my projects and not end up fighting with the implementation provided by the likes of bootstrap.

A closing note: I aim to try to build a small/medium project using Tailwind and give it a proper go. I found an article: Angular 8 + Tailwind CSS Guide by Sean Kerwin extremely useful in getting tailwind set up to be used inside of Angular.

Spring Shell Application: Purge History

James McNee — Sun, 15 Dec 2019 00:00:00 +0000

Hello all! It's been a while. I recently had the desire to create a shell application for a project that I am planning on working on; I wanted to create a small proof of concept application first so as to not complicate things. The first few hours of my forray into writing shell based apps was spent choosing a langage, and after much deliberation I decided to stick to what I know; Java and Spring. I am still yet to decide if this was a good idea; it certainly allowed me to be very productive and get a working prototype up in a reasonable amount of time, but I may have taken the easy road by sticking with Java.

Anyway, enough of the pre-amble. It turns out that Spring as always had my back when it came to creating a shell application. No need for manually registering user input and maintaining the app's lifecycle etc, Spring took care of all that, along with allowing me to use Dependency Injection and all the other good stuff spring offers. All of this came bundled up inside of the Spring shell library, or more accurately spring-shell-starter.

Spring shell provides some functionality by default, one of which is history; the ability to retain the commands entered by the user for later recall. This feature is extremely useful, and even persists across sessions, history can be recalled simply by typing history into the prompt. The issue is, there is no built in way to purge this history, which I deemed a neccessity for the application I was creating (see bottom of this article for a description).

After a bit of digging I found that the History functionality is provided by the org.jline package. This is all set up automagically by Spring for us, which is both extremely useful, and also very tedious to change the fucntionality of. There is little documentatin on how Spring is using jline so it took some trial and error to find the class that I needed to interact with. The class in question is LineReader: org.jline.reader.LineReader. There is an instance method on this class getHistory() which provides a history object to which the purge() method can be called to wipe it. Simple...

There is an issue when wiring in the LineReader bean as this creates a cyclical dependency. I got around this by marking the bean as Lazy in my components constructor (@Lazy LineReader lineReader). This means that it will not be wired in until it is needed, and when it is wired in, a proxy is wired, rather than the real object.

So why did I write this post... I found it tedious to find out how to implement this functionality, and if I can help someone else out then it was worth it!

Here is a sample of the code required:

import org.jline.reader.LineReader;
import org.jline.utils.AttributedString;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Lazy;
import org.springframework.shell.standard.ShellComponent;
import org.springframework.shell.standard.ShellMethod;
import org.springframework.shell.standard.ShellOption;

import java.io.IOException;

@ShellComponent
public class OtherCommands {

    private final LineReader lineReader;

    @Autowired
    public OtherCommands(@Lazy LineReader lineReader) {
        this.lineReader = lineReader;
    }

    @ShellMethod(value = "Clear all search history")
    public AttributedString clearHistory() throws IOException {
        lineReader.getHistory().purge();

        return CommandHelper.standardPrefixed("Info", "Command history has now been purged!");
    }
}

For those interested, the project that I wanted this for was an Urban Dictionary definition CLI. I was looking for a public API to play with while I was learning about Spring Shell and that seemed like a fun project; spoiler alert... it was. You can view the code for this, download the binary or fork it for yourself over on GitHub!

Google Analytics click events in Angular 2+ (the easy way)

James McNee — Mon, 29 Apr 2019 00:00:00 +0000

I recently wanted to add Google Analytics into my portfolio page and blog to track engagement and interaction. I did not need anything fancy basically just page views and button/link clicks. I started as I usually do by doing a quick google, something similar to 'Google Analytics Angular' and found a good article from Kissa Eric on Scotch.io (unfortunately the page has now been removed) with a detailed breakdown on how exactly GA can be added to an Angular project.

It basically boils down to adding the javascript provided by Google to the index.html file then creating an Angular service to send GA tracking events. As mentioned, this article was really useful in helping me to add Google Analytics to my project. The one area I think that was lacking however was in the way that it is recommended to send events on button clicks etc. The method provided is very programatic, binding to the click event of the element and then passing data to the service like this:

// Template
<input type="button" value="Click Me!" (click)="trackButton()" />

// Component
constructor(private analyticsService: AnalyticsService){}

trackButton() {
    this.analyticsService.emitEvent("category", "action", "label", "value");
}

While this method certainly works, it means that you are either requied to create a method for each click, or pass arguments into one method; either option is not ideal. In an ideal circumstance we would not have to edit the component at all. This is why I decided to create a directive that can fulfil this task. See below:

import { Directive, Input, HostListener } from '@angular/core';
import { GoogleAnalyticsService } from './google-analytics.service';

@Directive({
  selector: '[app-analytics]'
})
export class AnalyticsDirective {

  @Input("anayticsCategory") category: string;
  @Input("anayticsAction") action: string;
  @Input("anayticsLabel") label: string;
  @Input("anayticsvalue") value: number;

  constructor(private _analyticsService: GoogleAnalyticsService) {
  }

  @HostListener('click') onClick() {
    this._analyticsService.emitEvent(this.category, this.action, this.label, this.value);
  }
}

This directive allows me to tag an input or any element really and have click tracking applied; all without having to add any code to the componet itself. An example of how to add this in the template:

<li (click)="gotoHome()" app-analytics anayticsCategory="navigation" 
    anayticsAction="click" anayticsLabel="header-home-link">Home</li>

As you can see I have added the directive to the list item app-analytics, I use 'navigation' as my analyticsCategory to allow grouping of all navigation events in GA. I then set the analyticsAction and analyticsLabel.

In summary, following the aforementioned tutorial on Scotch.io (again, unfortunately the page has been removed), along with my directive, you can create a powerful way of tracking click events on any element. With a little tweaking you could also track other events like hovering etc.

Using GitLab CI to build and publish an Angular app [FTP]

James McNee — Mon, 22 Apr 2019 00:00:00 +0000

After recently deciding to update my portfolio page I thought it was time that I moved (slightly) towards a more professional automated publishing mechanism; for context my previous method was to directly upload a site via FTP. This called for... a pipeline!

Having used GoCD at my place of employment, I already had a good understanding of how a CI/CD (Continuous Integration/Delivery) system worked. My requirements for this pipeline were simple, build my angular app (using node) and publish the compiled project (via FTP) to my web host. I decided that as part of my pipeline I would also like a preprod stage, so that I can view my site in a production like environment before it went live.

While GoCD is a decent piece of software it felt a bit heavy for what I needed. I didn't want to set up a CI/CD environment for all my projects, I just wanted a single pipeline. I remembered that after I made the move a few months ago from GitHub to GitLab, there was a pipeline feature built in. It was a pleasant surprise when I discovered that it was also 100% free.

The setup process for the pipeline was different from what I have seen before; it uses a config file, rather than a GUI based solution. However this was fine, so I did a bit of Googling and found a good starting point on how to configure a pipeline with a node image. It was suprisingly intuitive to setup, first I setup a cache that I would shove the node_modules directory into and also the dist folder which would cache images and the like. This will save time with subsiquent builds by using cached resources rather than redownloading them from NPM.

cache:
  key: "$CI_BUILD_REF_NAME"
  untracked: true
  paths:
    - node_modules/
    - dist/

After this I needed to define the stages for my job, as mentioned I wanted to build, publish to a preprod environment and then finally to a production one.

stages:
  - build
  - preprod
  - prod

Next was to setup the first stage which is of course the build stage. Here my angular code will be compiled and bundled together.

runBuild:
  image: node:latest # Use the node image (has node installed)
  before_script:
   - npm install
  stage: build
  script:
    - npm run-script build:prod

Now I needed to copy the compiled source onto my FTP server in the preprod directory. This was where it got slightly more complicated as

I needed to abstract my FTP login details so to not store them in plain text on git.
I needed to understand how to use a CLI to copy files via FTP

After some more Googling and browsing StackOverflow I found a useful post describing the various options of the CLI; with this and another helpful post on the GitLab forum I created a preprod job:

runPublishPreProd:
  stage: preprod
  before_script: 
    - apt-get update
    - apt-get install -y lftp # Install the FTP library
  artifacts:
    name: "$CI_BUILD_NAME/$CI_BUILD_REF_NAME" # Using the artifact from the previous stage
    paths:
    - dist/ # Path inside the artifact bundle
    expire_in: 2d # Keep around for 2 days
  script:
    - cd dist/portfolio/ # Change into the dist/portfolio directory (that came from the build stage)
    - lftp -u $FTP_USERNAME,$FTP_PASSWORD $FTP_HOST -e "mirror -e -R -p ./ preprod-main-site ; quit"
    - echo "Deployment Completed!"
  dependencies:
    - runBuild # This stage cannot be run until the runBuild task has been completed.

This was it, very simple. Now it was time to do the prod stage, which was almost identical except that I needed to add a manual trigger; this allows me to verify in preprod before pushing live. It was as simple as adding the following to the end of the task:

  when: 
    manual

Overall a pretty decent experience. I now have a pipeline that is triggered by pushing to the git repository and auto deploys into preprod. Hopefully this post will be of use to anyone who is wanting to do a similar thing!

Hello World

James McNee — Mon, 22 Apr 2019 00:00:00 +0000

It seems only fitting that the first post on this blog should be around the journey that I have taken to create it. Not only create the blog section but the whole portfolio site. It will be useful to provide some context to start off with around why I decided to create the site and what I had before. Around 2 years ago I decided to create a site to showcase my personal projects and also have a feed of what I have been looking at and watching recently. This was my main site until a few days ago when a new incarnation went live. It was clear that a number of issues were present in the old site; both in design and purpose. Responsiveness was completely lacking from the site; this meant that it looked awful on mobile devices. Another issue came from the only feed being sourced from a YouTube playlist, this restricted my ability to share articles and talks that I have been interested in that was outside of YouTube.

Therefore it was clear, out with the old; in with the new. Learning from my mistakes of the past, it was obvious that I needed to start with a CSS framework that would allow me to build upon a strong foundation. In the past I have tried to build the entire template from lab and realistically CSS is not my strongest skill; not my weakest, but certainly I am more comfortable supplimenting an existing base. I decided Bootstrap would be my new best friend, and started to sketch out a design with the trusty old pen and paper. It was in this phase that I decided to look around for inspiration and came across a great looking bootstrap theme by Jeromelachaud and decided to base my design around it. I chose not to use the template itself as this would allow me greater control and flexibility when adding new components; using the template would also not have allowed me to properly structure towards to the Angular framework.

A few sketches later I set about creating the site, with some frustration here and magic there, it was born; a brand new professional portfolio page. I also decided to integrate with GitLab's CI to provide continuous integration and automatic deployment with both a preprod and prod stage to boot; more on this in another blog post though! So there it was, all done and dusted; brand new site; automated pipeline, but I felt there was still something missing. Dynamic content. I decided to give blogging a try; after having tried it in the past and gave up after a few posts. This time though I have more experience under my belt and a greater appreciation for the sharing of knowledge. I decided to create my own simple interface and service to provide the posts from a MySQL DB, and even markdown integration!

Anyway, that is all for now. First post done! I am hoping to write a piece around my setup with the GitLab CI and hopefully provide some useful bits for anyone trying a similar endeavour

James McNee - Blog

Fascinating Faceting with Postgres

What on earth is a facet?

The rules of the facets

Faceting in Postgres

Conclusion

Managing permissions with Postgres event triggers

Setting the scene

Provisioning and management

Custom roles

Two problems to overcome

Perpetual permissions

An ordering problem

Event Triggers to the rescue

What is an event trigger?

Defining the event trigger and function

Reconciliation

Conclusion

Waiting is hard... but just Persevere(JS)

Preface

Shallow Integration Testing

Preventing Catastrophe using Gum

It's all about context

Visual signals

Custom Prompts

Colour Coded Shells

Gumming it up

What is Gum?

Creating a kubectl wrapper

Ensuring that the wrapper is invoked

Summary

Hardening Security: From Zero to Hero

Preface

But what makes a site 'secure'?

So what was wrong?

Implementing a Content Security Policy (CSP)

A brief note on the <meta> tag

Why would inline scripts and styles be a problem?

A brief note on CSP reporting

Building the CSP

Implementing HTTP Strict Transport Security (HSTS)

Implementing X-Frame-Options (XFO)

Summarising

Adding dynamic search to a static Eleventy site

Switching to Eleventy

Searching for a way to search

Building it from lab

Creating a search payload

Creating the markup

Implementing the dynamic results

Tying it all together

Summary

Gradle Plugin: Adding an external dependency

Reverse Proxies: Take care what is being exposed!

Sorry... It's time for the other shoe to drop

I'm convinced! How can we solve this?

To PATCH or not to PATCH; that is not the question

How a PUT request works

How a PATCH request works

The 'Merge Patch' implementation

The 'JSON Patch' implementation

How does the server know the difference...

Summary

Preventing log spam in Java

Propelled by a Tailwind

Spring Shell Application: Purge History

Google Analytics click events in Angular 2+ (the easy way)

Using GitLab CI to build and publish an Angular app [FTP]

Hello World

A brief note on the `<meta>` tag