Tyler Kennedy

Building a Black Hole Shader in Godot 4

2026-01-15T00:00:00-05:00

My upcoming hobby 4X game takes place over an entire galaxy, and what's at the center of every galaxy? A supermassive black hole. So, we need to make one.

This isn't a physically accurate simulation. We're building a stylized effect that looks right, using simplified physics approximations. The real math behind Gargantua involved over a year of work by Kip Thorne and the DNEG VFX team. Ours will take about 400 lines of shader code.

Stage 1: Sphere Intersection

Every raymarch shader needs to know where the ray enters and exits the volume. We use standard ray-sphere intersection math:

vec2 intersect_sphere(vec3 ray_origin, vec3 ray_dir, vec3 sphere_centre, float radius) {
    vec3 offset = ray_origin - sphere_centre;
    float b = 2.0 * dot(offset, ray_dir);
    float c = dot(offset, offset) - radius * radius;
    float discriminant = b * b - 4.0 * c;

    if (discriminant > 0.0) {
        float s = sqrt(discriminant);
        float dst_near = max(0.0, (-b - s) * 0.5);
        float dst_far = (-b + s) * 0.5;
        if (dst_far >= 0.0) {
            return vec2(dst_near, dst_far - dst_near);
        }
    }
    return vec2(MAX_FLOAT, 0.0);
}

This returns two values: the distance to where the ray enters the sphere, and how far it travels through it. If the ray misses entirely, we return a sentinel value.

The vertex shader is minimal - we just extract the world position and object scale for use in the fragment shader:

void vertex() {
    frag_pos_ws = (MODEL_MATRIX * vec4(VERTEX, 1.0)).xyz;
    centre = MODEL_MATRIX[3].xyz;
    object_scale = vec3(
        length(MODEL_MATRIX[0].xyz),
        length(MODEL_MATRIX[1].xyz),
        length(MODEL_MATRIX[2].xyz)
    );
}

Stage 2: Raymarching Foundation

With sphere intersection working, we can step through it. The basic loop looks like this:

vec3 current_ray_pos = ray_origin + ray_dir * sphere_entry_distance;
vec3 current_ray_dir = ray_dir;

for (int i = 0; i < ray_steps; i++) {
    // Do something at each step
    current_ray_pos += current_ray_dir * step_size;

    // Exit if we've left the sphere
    if (length(current_ray_pos - centre) > sphere_radius) {
        break;
    }
}

The heatmap shows how many steps each pixel required. More steps near the edges where the ray travels further through the sphere. This is why ray_steps is a key performance knob - too few and you get banding artifacts, too many and your GPU melts.

Stage 3: Gravitational Lensing

Here's where the magic happens. At each step, we bend the ray toward the center using an inverse-square force (mimicking gravity):

vec3 dir_to_centre = centre - current_ray_pos;
float dst_to_centre = length(dir_to_centre);
dir_to_centre /= dst_to_centre;  // Normalize

float norm_dst = max(dst_to_centre / sphere_radius, 1e-4);
float force = g_const / (norm_dst * norm_dst);  // Inverse-square
current_ray_dir = normalize(current_ray_dir + dir_to_centre * force * step_size);

The g_const uniform controls how strongly gravity bends light. Higher values create more extreme lensing. The force follows an inverse-square law (1/r²), increasing rapidly as you approach the center - just like real gravity.

Notice how the stars warp around the center. Light rays passing close to the black hole bend dramatically, while distant rays barely deflect. With perfect alignment, this creates an Einstein ring - but any viewing angle shows gravitational lensing.

Stage 4: The Event Horizon

At some radius, the gravitational pull is so strong that nothing escapes. We model this as simple absorption:

if (dst_to_centre <= event_horizon_radius) {
    black_hole_mask = 1.0;
    break;
}

To avoid hard aliased edges, we add a soft falloff zone around the horizon:

float falloff_zone = event_horizon_radius * 1.5;
if (curr_dst_to_centre < falloff_zone) {
    float proximity = 1.0 - (curr_dst_to_centre - event_horizon_radius)
                          / (falloff_zone - event_horizon_radius);
    black_hole_mask = max(black_hole_mask, clamp(proximity, 0.0, 1.0));
}

The black_hole_mask accumulates as rays get close to the horizon, creating a smooth transition into darkness rather than a sharp circle.

Stage 5: Detecting the Disc

The accretion disc is modeled as an infinitely thin plane perpendicular to the black hole's up vector. We detect when a ray crosses this plane by checking for a sign change:

float prev_plane_dist = dot(prev_ray_pos - centre, disc_dir);
float curr_plane_dist = dot(current_ray_pos - centre, disc_dir);

if (prev_plane_dist * curr_plane_dist < 0.0) {
    // Ray crossed the disc plane this step
    float interp = abs(prev_plane_dist) / (abs(prev_plane_dist) + abs(curr_plane_dist));
    vec3 crossing_pos = mix(prev_ray_pos, current_ray_pos, interp);
}

The linear interpolation finds the exact crossing point between our two sample positions. This is more accurate than just using the current position, especially with larger step sizes.

We also check that the crossing point falls within the disc's inner and outer radii. The inner edge exists because matter too close to the horizon falls in rather than orbiting.

Stage 6: Disc Temperature Gradient

Real accretion discs are hotter near the center where matter is compressed and moving faster. We model this with a radial color gradient:

vec2 disc_uv(vec3 planar_disc_pos, float radius) {
    float sample_dist = length(planar_disc_pos) / max(radius, 1e-6);
    float phi = atan(planar_disc_pos.z, planar_disc_pos.x);
    phi = phi / (2.0 * PI) + 0.5;
    return vec2(sample_dist, phi);
}

The UV coordinates give us radial distance (u) and angular position (v). We use the radial coordinate to blend between a hot inner color (white-yellow) and a cooler outer color:

vec3 inner_color = vec3(1.0, 1.0, 0.95);
float radial_t = clamp(uv.x, 0.0, 1.0);
vec3 temp_color = mix(inner_color, base_color, pow(radial_t, 0.6));

The pow(radial_t, 0.6) gives us a non-linear falloff - the hot inner region stays bright longer before transitioning to the outer color.

Stage 7: Doppler Beaming

When matter orbits at relativistic speeds, the side moving toward you appears brighter than the side moving away. This is called Doppler beaming, and it's why real black hole images show asymmetric brightness.

We simulate this by comparing distances after a small rotation:

vec3 rotated_pos = rotate_about_axis(planar_disc_pos, disc_dir, 0.01);
float doppler_dist = (length(rotated_pos - camera_pos) - length(planar_disc_pos - camera_pos))
                     / safe_radius;
float doppler_contribution = doppler_dist * disc_speed * doppler_beaming_factor;

The approaching side gets significantly brighter. Combined with the temperature gradient, this creates the distinctive asymmetric look of accretion discs.

Stage 8: Procedural Turbulence

A perfectly smooth disc looks artificial. Real accretion discs have turbulent flow patterns. We generate these procedurally using Fractional Brownian Motion (FBM) - layered noise at different frequencies:

float seamless_fbm_cylinder(float u, float v, float scale, float time, int octaves) {
    // Convert angular coordinate to circle to avoid seams
    float angle = v * 2.0 * PI;
    vec3 cyl_pos = vec3(u * scale, cos(angle) * scale * 0.5, sin(angle) * scale * 0.5);
    cyl_pos.x += time;  // Animate

    float value = 0.0;
    float amplitude = 0.5;
    float frequency = 1.0;

    for (int i = 0; i < octaves; i++) {
        float n1 = noise2d(cyl_pos.xy * frequency);
        float n2 = noise2d(cyl_pos.xz * frequency + 100.0);
        value += amplitude * mix(n1, n2, 0.5);
        amplitude *= 0.5;
        frequency *= 2.0;
    }
    return value;
}

The trick here is using cylindrical coordinates. If we just used 2D noise with the angular coordinate directly, we'd get a visible seam where 0 meets 2π. By wrapping the angle onto a circle in 3D space, the noise tiles seamlessly.

The second image shows turbulent displacement applied - we use one layer of FBM to distort the coordinates of another, creating swirling patterns.

Stage 9: Angular Streaks

For extra visual interest, we add radial streaks that spiral around the disc:

float soft_edge(float radius_norm, float angle, float time) {
    float scaled_angle = angle * streak_count;
    float edge_noise = seamless_noise1d(scaled_angle + time * 0.1, streak_count * period);
    float falloff = 1.0 - smoothstep(falloff_start, 1.0, radius_norm);
    return falloff;
}

These create the appearance of density variations in the orbiting material.

Stage 10: Screen-Space Lensing

After raymarching, we have a bent ray direction. We project this back to screen space to find where to sample the background:

vec3 distorted_ray_dir = has_sphere ? current_ray_dir : ray_dir;
vec4 ray_camera_space = VIEW_MATRIX * vec4(distorted_ray_dir, 0.0);
vec4 ray_uv_projection = PROJECTION_MATRIX * vec4(ray_camera_space.xyz, 0.0);
vec2 distorted_screen_uv = ray_uv_projection.xy * 0.5 + 0.5;

We blend based on how much the ray actually bent, creating smooth transitions at the sphere boundary:

float ray_bend = 1.0 - dot(ray_dir, current_ray_dir);
float bend_blend = smoothstep(0.0, 0.15, ray_bend);
distorted_screen_uv = mix(screen_uv, distorted_screen_uv, bend_blend);

This is the full gravitational lensing effect - the background warps dramatically near the event horizon.

Putting It Together

The final composition blends the lensed background with the disc emission:

vec3 background_col = texture(screen_texture, distorted_screen_uv).rgb * (1.0 - black_hole_mask);
vec3 final_disc_col = calc_disc_color(...);
disc_hit *= tex_col * disc_color.a * streaks * horizon_fade * radial_falloff;
vec3 col = mix(background_col, final_disc_col, disc_hit);

Full Shader

/*
Copyright 2026 Tyler Kennedy

Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the “Software”), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Procedural Black Hole Shader
============================

Mental model:
- Raymarch inside a bounding sphere, bending the ray toward the center.
- If the ray hits the event horizon, it is absorbed; if it hits the disc, it emits.
- The final ray direction warps the screen sample to create lensing.

This is a stylized effect with simple physics approximations.
*/

shader_type spatial;
render_mode unshaded, cull_disabled, depth_draw_never;

uniform sampler2D disc_texture : filter_linear, repeat_enable;
uniform float disc_outer_radius : hint_range(0.0, 1.0) = 1.0;
uniform float disc_inner_radius : hint_range(0.0, 1.0) = 0.25;
uniform float disc_speed : hint_range(0.0, 10.0) = 2.0;
uniform vec4 disc_color : source_color = vec4(1.0, 0.0, 0.0, 1.0);
uniform float doppler_beaming_factor : hint_range(0.0, 200.0) = 66.0;
uniform float hue_radius : hint_range(0.0, 1.0) = 0.75;
uniform float hue_shift_factor : hint_range(-1.0, 1.0) = -0.03;
uniform float streak_intensity : hint_range(0.0, 1.0) = 0.5;
uniform float streak_count : hint_range(1.0, 50.0) = 20.0;

uniform float turbulence_intensity : hint_range(0.0, 1.0) = 0.3;
uniform float turbulence_scale : hint_range(0.5, 10.0) = 3.0;
uniform float turbulence_speed : hint_range(0.0, 2.0) = 0.5;
uniform int turbulence_octaves : hint_range(1, 6) = 4;

uniform int ray_steps : hint_range(32, 512) = 256;
uniform float step_size : hint_range(0.001, 1.0) = 0.1;
uniform float ss_radius : hint_range(0.0, 1.0) = 0.1;
uniform float g_const : hint_range(0.0, 2.0) = 0.3;

uniform sampler2D screen_texture : hint_screen_texture, filter_linear_mipmap;

const float MAX_FLOAT = 3.402823466e+38;
const float SEAM_BLEND_START = 0.9;

varying vec3 frag_pos_ws;
varying vec3 centre;
varying vec3 object_scale;

void vertex() {
    frag_pos_ws = (MODEL_MATRIX * vec4(VERTEX, 1.0)).xyz;
    centre = MODEL_MATRIX[3].xyz;
    object_scale = vec3(
        length(MODEL_MATRIX[0].xyz),
        length(MODEL_MATRIX[1].xyz),
        length(MODEL_MATRIX[2].xyz)
    );
}

float hash(float n) { return fract(sin(n) * 43758.5453); }

float noise1d(float x) {
    float i = floor(x);
    float f = fract(x);
    f = f * f * (3.0 - 2.0 * f);
    return mix(hash(i), hash(i + 1.0), f);
}

float noise2d(vec2 p) {
    vec2 i = floor(p);
    vec2 f = fract(p);
    f = f * f * (3.0 - 2.0 * f);

    float a = hash(dot(i, vec2(1.0, 157.0)));
    float b = hash(dot(i + vec2(1.0, 0.0), vec2(1.0, 157.0)));
    float c = hash(dot(i + vec2(0.0, 1.0), vec2(1.0, 157.0)));
    float d = hash(dot(i + vec2(1.0, 1.0), vec2(1.0, 157.0)));

    return mix(mix(a, b, f.x), mix(c, d, f.x), f.y);
}

float seamless_fbm_cylinder(float u, float v, float scale, float time, int octaves) {
    float angle = v * 2.0 * PI;
    vec3 cyl_pos = vec3(u * scale, cos(angle) * scale * 0.5, sin(angle) * scale * 0.5);
    cyl_pos.x += time;

    float value = 0.0;
    float amplitude = 0.5;
    float frequency = 1.0;

    for (int i = 0; i < octaves; i++) {
        float n1 = noise2d(cyl_pos.xy * frequency);
        float n2 = noise2d(cyl_pos.xz * frequency + 100.0);
        value += amplitude * mix(n1, n2, 0.5);
        amplitude *= 0.5;
        frequency *= 2.0;
    }
    return value;
}

vec2 turbulent_displacement_seamless(float u, float v, float time, float scale, float intensity, int octaves) {
    vec2 offset = vec2(
        seamless_fbm_cylinder(u, v, scale, time * 0.7, octaves),
        seamless_fbm_cylinder(u, v + 0.37, scale, time * 0.5, octaves)
    );
    return (offset - 0.5) * intensity;
}

float seamless_noise1d(float x, float period) {
    float t = fract(x / period);
    float n1 = noise1d(x);
    float n2 = noise1d(x - period);
    return mix(n1, n2, smoothstep(SEAM_BLEND_START, 1.0, t));
}

float soft_edge(float radius_norm, float angle, float time) {
    float period = 2.0 * PI;
    float scaled_angle1 = angle * streak_count;
    float scaled_angle2 = angle * streak_count * 2.3 + 100.0;

    float edge_noise = seamless_noise1d(scaled_angle1 + time * 0.1, streak_count * period) * 0.5 + 0.5;
    float edge_noise2 = seamless_noise1d(scaled_angle2 + time * 0.15, streak_count * 2.3 * period) * 0.5 + 0.5;
    float combined_noise = edge_noise * 0.6 + edge_noise2 * 0.4;

    float falloff_start = 0.6 + combined_noise * streak_intensity * 0.4;
    float falloff = 1.0 - smoothstep(falloff_start, 1.0, radius_norm);

    return falloff;
}

vec2 intersect_sphere(vec3 ray_origin, vec3 ray_dir, vec3 sphere_centre, float radius) {
    vec3 offset = ray_origin - sphere_centre;
    float b = 2.0 * dot(offset, ray_dir);
    float c = dot(offset, offset) - radius * radius;
    float discriminant = b * b - 4.0 * c;

    if (discriminant > 0.0) {
        float s = sqrt(discriminant);
        float dst_near = max(0.0, (-b - s) * 0.5);
        float dst_far = (-b + s) * 0.5;
        if (dst_far >= 0.0) {
            return vec2(dst_near, dst_far - dst_near);
        }
    }
    return vec2(MAX_FLOAT, 0.0);
}

float remap(float v, float min_old, float max_old, float min_new, float max_new) {
    float denom = max_old - min_old;
    if (abs(denom) < 1e-6) {
        return min_new;
    }
    return min_new + (v - min_old) * (max_new - min_new) / denom;
}

vec2 disc_safe_radii() {
    float safe_outer = max(disc_outer_radius, 1e-4);
    float safe_inner = clamp(disc_inner_radius, 0.0, safe_outer - 1e-4);
    return vec2(safe_inner, safe_outer);
}

vec2 disc_uv(vec3 planar_disc_pos, float radius) {
    float sample_dist = length(planar_disc_pos) / max(radius, 1e-6);
    float phi = atan(planar_disc_pos.z, planar_disc_pos.x);
    phi = phi / (2.0 * PI) + 0.5;
    return vec2(sample_dist, phi);
}

vec3 linear_to_gamma(vec3 col) {
    col = max(col, vec3(0.0));
    return max(1.055 * pow(col, vec3(0.416666667)) - 0.055, vec3(0.0));
}

vec3 gamma_to_linear(vec3 col) {
    return col * (col * (col * 0.305306011 + 0.682171111) + 0.012522878);
}

vec3 hdr_intensity(vec3 col, float intensity) {
    col = linear_to_gamma(col);
    col *= pow(2.0, intensity);
    col = gamma_to_linear(col);
    return col;
}

vec3 rgb_to_hsv(vec3 c) {
    vec4 K = vec4(0.0, -1.0 / 3.0, 2.0 / 3.0, -1.0);
    vec4 p = mix(vec4(c.bg, K.wz), vec4(c.gb, K.xy), step(c.b, c.g));
    vec4 q = mix(vec4(p.xyw, c.r), vec4(c.r, p.yzx), step(p.x, c.r));
    float d = q.x - min(q.w, q.y);
    float e = 1.0e-10;
    return vec3(abs(q.z + (q.w - q.y) / (6.0 * d + e)), d / (q.x + e), q.x);
}

vec3 hsv_to_rgb(vec3 c) {
    vec4 K = vec4(1.0, 2.0 / 3.0, 1.0 / 3.0, 3.0);
    vec3 p = abs(fract(c.xxx + K.xyz) * 6.0 - K.www);
    return c.z * mix(K.xxx, clamp(p - K.xxx, 0.0, 1.0), c.y);
}

vec3 rotate_about_axis(vec3 v, vec3 axis, float rotation) {
    float s = sin(rotation);
    float c = cos(rotation);
    float omc = 1.0 - c;
    axis = normalize(axis);
    mat3 rot = mat3(
        vec3(omc * axis.x * axis.x + c, omc * axis.x * axis.y + axis.z * s, omc * axis.z * axis.x - axis.y * s),
        vec3(omc * axis.x * axis.y - axis.z * s, omc * axis.y * axis.y + c, omc * axis.y * axis.z + axis.x * s),
        vec3(omc * axis.z * axis.x + axis.y * s, omc * axis.y * axis.z - axis.x * s, omc * axis.z * axis.z + c)
    );
    return rot * v;
}

vec3 calc_disc_color(vec3 base_color, vec3 planar_disc_pos, vec3 disc_dir, vec3 camera_pos, float u, float v, float radius, float time) {
    float radial_t = clamp(u, 0.0, 1.0);

    float turb_time = time * turbulence_speed;
    float turb_noise = seamless_fbm_cylinder(u * 2.0, v, turbulence_scale, turb_time, turbulence_octaves);

    vec2 swirl_offset = turbulent_displacement_seamless(u * 2.0, v, turb_time, turbulence_scale * 0.5, turbulence_intensity * 0.3, turbulence_octaves);
    float swirl_noise = seamless_fbm_cylinder(u * 2.0 + swirl_offset.x, v + swirl_offset.y, turbulence_scale * 1.5, turb_time * 0.5, turbulence_octaves);

    float combined_turb = mix(turb_noise, swirl_noise, 0.5);
    float turb_intensity_mod = (combined_turb - 0.5) * 2.0 * turbulence_intensity;

    float hot_spot = pow(max(0.0, combined_turb - 0.6) * 2.5, 2.0) * turbulence_intensity;

    vec3 inner_color = vec3(1.0, 1.0, 0.95);
    vec3 temp_color = mix(inner_color, base_color, pow(radial_t, 0.6));

    temp_color = mix(temp_color, inner_color, hot_spot * 0.5);

    float base_intensity = remap(radial_t, 0.0, 1.0, 1.35, -2.5);

    vec3 rotated_pos = rotate_about_axis(planar_disc_pos, disc_dir, 0.01);
    float safe_radius = max(radius, 1e-6);
    float doppler_dist = (length(rotated_pos - camera_pos) - length(planar_disc_pos - camera_pos)) / safe_radius;
    float doppler_contribution = doppler_dist * disc_speed * doppler_beaming_factor * (1.0 - radial_t * 0.7);

    float intensity = base_intensity + doppler_contribution + turb_intensity_mod + hot_spot * 1.5;
    intensity = clamp(intensity, -3.0, 3.0);

    vec3 new_color = hdr_intensity(temp_color, intensity);

    vec3 hsv = rgb_to_hsv(new_color);
    float hue_shift = clamp(remap(radial_t, hue_radius, 1.0, 0.0, 1.0), 0.0, 1.0);
    hue_shift += (combined_turb - 0.5) * turbulence_intensity * 0.1;
    hsv.x += hue_shift * hue_shift_factor;
    new_color = hsv_to_rgb(hsv);

    return new_color;
}

void fragment() {
    vec3 ray_origin = CAMERA_POSITION_WORLD;
    vec3 ray_dir = normalize(frag_pos_ws - ray_origin);

    float sphere_radius = 0.5 * min(min(object_scale.x, object_scale.y), object_scale.z);
    vec2 outer_sphere = intersect_sphere(ray_origin, ray_dir, centre, sphere_radius);
    bool has_sphere = outer_sphere.x < MAX_FLOAT;

    vec3 disc_dir = normalize((MODEL_MATRIX * vec4(0.0, 1.0, 0.0, 0.0)).xyz);
    vec2 radii = disc_safe_radii();
    float disc_radius = sphere_radius * radii.y;
    float inner_radius = sphere_radius * radii.x;
    float scaled_step = step_size * sphere_radius;

    float disc_hit = 0.0;
    float black_hole_mask = 0.0;
    vec3 sample_pos = vec3(MAX_FLOAT, 0.0, 0.0);
    vec3 current_ray_pos = has_sphere ? (ray_origin + ray_dir * outer_sphere.x) : (ray_origin + ray_dir);
    vec3 current_ray_dir = ray_dir;
    float event_horizon_radius = ss_radius * sphere_radius;

    if (has_sphere) {
        for (int i = 0; i < ray_steps; i++) {
            vec3 dir_to_centre = centre - current_ray_pos;
            float dst_to_centre = length(dir_to_centre);
            if (dst_to_centre <= event_horizon_radius) {
                black_hole_mask = 1.0;
                break;
            }
            dir_to_centre /= dst_to_centre;

            if (dst_to_centre > sphere_radius + scaled_step) {
                break;
            }

            vec3 prev_ray_pos = current_ray_pos;
            float prev_plane_dist = dot(prev_ray_pos - centre, disc_dir);

            float norm_dst = max(dst_to_centre / sphere_radius, 1e-4);
            float force = g_const / (norm_dst * norm_dst);
            current_ray_dir = normalize(current_ray_dir + dir_to_centre * force * step_size);

            current_ray_pos += current_ray_dir * scaled_step;

            float curr_dst_to_centre = length(current_ray_pos - centre);

            float falloff_zone = event_horizon_radius * 1.5;
            if (falloff_zone > event_horizon_radius + 1e-6 && curr_dst_to_centre < falloff_zone) {
                float proximity = 1.0 - (curr_dst_to_centre - event_horizon_radius) / (falloff_zone - event_horizon_radius);
                black_hole_mask = max(black_hole_mask, clamp(proximity, 0.0, 1.0));
            }

            if (curr_dst_to_centre <= event_horizon_radius) {
                black_hole_mask = 1.0;
                break;
            }

            float curr_plane_dist = dot(current_ray_pos - centre, disc_dir);
            if (disc_hit < 1.0 && prev_plane_dist * curr_plane_dist < 0.0) {
                float interp = abs(prev_plane_dist) / (abs(prev_plane_dist) + abs(curr_plane_dist));
                vec3 crossing_pos = mix(prev_ray_pos, current_ray_pos, interp);
                vec3 planar_offset = crossing_pos - centre - dot(crossing_pos - centre, disc_dir) * disc_dir;
                float radial_dist_sqr = dot(planar_offset, planar_offset);
                float outer_fade_radius = disc_radius * 1.3;
                if (radial_dist_sqr < outer_fade_radius * outer_fade_radius) {
                    disc_hit = 1.0;
                    sample_pos = crossing_pos;
                }
            }

            if (disc_hit > 0.0 && abs(curr_plane_dist) > abs(prev_plane_dist)) {
                break;
            }
        }
    }

    vec2 uv = vec2(0.0);
    vec3 planar_disc_pos = vec3(0.0);
    float streaks = 1.0;
    float radial_falloff = 1.0;
    if (sample_pos.x < MAX_FLOAT) {
        planar_disc_pos = sample_pos - dot(sample_pos - centre, disc_dir) * disc_dir - centre;
        uv = disc_uv(planar_disc_pos, disc_radius);
        uv.y += TIME * disc_speed * 0.1;

        float inner_norm = radii.x / radii.y;
        float radius_norm = clamp((uv.x - inner_norm) / max(1.0 - inner_norm, 1e-4), 0.0, 1.0);
        streaks = soft_edge(radius_norm, uv.y * PI * 2.0, TIME);

        float radial_dist = length(planar_disc_pos) / disc_radius;
        radial_falloff = 1.0 - smoothstep(0.85, 1.3, radial_dist);
    }
    float tex_col = texture(disc_texture, uv).r;

    vec2 screen_uv = SCREEN_UV;

    vec3 distorted_ray_dir = has_sphere ? current_ray_dir : ray_dir;
    vec4 ray_camera_space = VIEW_MATRIX * vec4(distorted_ray_dir, 0.0);
    vec4 ray_uv_projection = PROJECTION_MATRIX * vec4(ray_camera_space.xyz, 0.0);
    vec2 distorted_screen_uv = ray_uv_projection.xy * 0.5 + 0.5;

    float ray_bend = 1.0 - dot(ray_dir, current_ray_dir);
    float bend_blend = smoothstep(0.0, 0.15, ray_bend);

    float edge_fade_x = smoothstep(0.0, 0.15, min(screen_uv.x, 1.0 - screen_uv.x));
    float edge_fade_y = smoothstep(0.0, 0.15, min(screen_uv.y, 1.0 - screen_uv.y));

    float lens_blend = has_sphere ? bend_blend * edge_fade_x * edge_fade_y : 0.0;
    distorted_screen_uv = mix(screen_uv, distorted_screen_uv, lens_blend);

    vec3 background_col = texture(screen_texture, distorted_screen_uv).rgb * (1.0 - black_hole_mask);

    vec3 final_disc_col = calc_disc_color(disc_color.rgb, planar_disc_pos, disc_dir, ray_origin, uv.x, uv.y, disc_radius, TIME);

    float horizon_fade = 1.0;
    if (sample_pos.x < MAX_FLOAT) {
        float sample_dist_to_centre = length(sample_pos - centre);
        if (event_horizon_radius > 1e-6) {
            float fade_center = event_horizon_radius * 1.2;
            float fade_width = max(fwidth(sample_dist_to_centre) * 2.0, event_horizon_radius * 0.15);
            horizon_fade = smoothstep(fade_center - fade_width, fade_center + fade_width, sample_dist_to_centre);
        }
    }

    disc_hit *= tex_col * disc_color.a * streaks * horizon_fade * radial_falloff;
    vec3 col = mix(background_col, final_disc_col, disc_hit);

    ALBEDO = col;
    ALPHA = 1.0;
}

Night Sky: Procedural Skyboxes in Your Browser

2026-01-15T00:00:00-05:00

Night Sky generates high-resolution (2k-16k) nebula, night and day skyboxes entirely in your browser. No license, your skyboxes are your own. My upcoming hobby 4X is source-available, so I need as many assets as possible that I can make freely available for redistribution.

Performance

The rendering pipeline is written in Rust and compiled to WebAssembly. The browser divides the image into horizontal tiles and distributes them across multiple Web Workers, so rendering happens in parallel without freezing the UI. On browsers that support it, a SIMD-optimized build runs automatically for a bit of extra speed. We use fp16 end-to-end to get a 16k HDR to fit under the typical 2GB limit for a browser WASM sandbox.

At first the implementation was purely TypeScript, but doing the raymarched nebulas at 16k and 32k with pure TypeScript was absurdly slow. Plus, this way we can wrap the Rust module and make it available for other things.

Export Formats

Equirectangular PNG - Standard 2:1 panorama, works directly in most game engines and 3D software.
Cubemap ZIP - Six square faces for engines that prefer cube maps.
Radiance HDR - Full dynamic range for professional lighting. Stars can blow out past 1.0, giving proper bloom when used in a PBR renderer.

Try It

The tool is live at tkte.ch/night-sky. Hit randomize, find a seed you like, and export at whatever resolution you need.

Sunsetting The Minecraft Protocol Wiki and Notifico

2024-11-11T00:00:00-05:00

tl;dr I've decided to sunset two of my longest-running projects, wiki.vg and n.tkte.ch. Both of these 'temporary' projects have well exceeded their intended lifespan and are now over a decade old.

wiki.vg

wiki.vg started as a simple page on my personal DokuWiki in 2010 to document the Minecraft protocol and internals. Through the hard work of hundreds of contributors, it grew into a comprehensive resource for anyone looking to understand how Minecraft works. The IRC community the formed around it has been a great place to learn and share knowledge about Minecraft and has survived the rise and fall of numerous chat platforms and the epic collapse of freenode. By-and-large, this has been the most drama-free community I've ever been a part of and has essentially run without needing any moderation for over a decade.

To say that wiki.vg became a success is an understatement. It's had billions of page views, and was probably used by every single custom Minecraft project directly or indirectly. Even Mojang's own developers have used it as a reference and would occasionally make the rare contribution. Several early contributors went on to work for Mojang and Microsoft themselves. Millions of end-users have benefited from the knowledge shared on wiki.vg.

In the end, this project was always intended to be temporary. We were promised a modding API and documentation for Minecraft that never came, and the project was supposed to be a stopgap until that happened.

On November 30th, wiki.vg will close for good. The final content will be archived and made available for download in MediaWiki's XML export format under the CC-BY-SA license, along with an archive of images and other media. My hope is that the content will be subsumed into the official Minecraft wiki, but if not, it will be available for anyone to host.

FAQ

Q: Would you accept offers to take over responsibility for its hosting and upkeep?

A: No. I appreciate the offer, but I don't believe the user's that have signed up for an account on wiki.vg have given me the right to decide that their data should be transferred to someone else.

Q: What will happen with the domain?

A: The domain will be redirected to the archive of the content for the foreseeable future. The domain has trust behind it, and I don't want the headache of ensuring that any successor doesn't turn it into an abomination like they did with my first Minecraft-related website, minecraftskins.com.

n.tkte.ch

Before Discord or Slack existed, practically every open source project with more than a few contributors had an IRC channel on Freenode. We used svn, Jenkins, and dozens of other tools that have for the most part been replaced by GitHub and its ecosystem. One of the most important tools for monitoring all of these disparate systems was cia.vc, a service that would listen to commit hooks and other events and relay them to IRC.

Then, suddenly on September 26th 2012, cia.vc went down and never came back up. Poof. It was gone. Everyone scrambled to make a quick replacement, resulting in irker, kgb and my own notifico. Notifico was a simple service to consume webhooks, and at the time, a clone of cia.vc's XML-RPC payload and relay it to IRC. It really was intended to be temporary as many of us expected cia.vc to come back online, but it never did. It appears there were never any backups. A decade later, Notifico is still running and has delivered over 15 million messages for thousands of projects like CPython, FreeBSD, Godot, Qutebrowser, and even NASA weather balloons.

But, the project was never designed to be maintained long-term and its codebase really shows that, making refactoring and the addition of new features time-consuming. Tests were never even a consideration for a project that was supposed to be a quick hack.

Notifico is completely open source under the MIT license, and easy to host. The project itself will not disappear, but my personal instance which has been running for over a decade will be shut down on November 30th.

Why?

These services were maintained ad-free and cost-free for over a decade, but it's not really the material cost that's driving this decision. These projects were never meant to be permanent, and even though IRC never really changes, there's always a maintenance burden that comes with running a service. Keeping servers up to date, ensuring emails are getting delivered, dealing with deprecated and disappearing dependencies, user issues, and so on. Allowing public registration will always lead to spam at some point which is a whole other can of worms.

As I get older, I find myself with less and less time to dedicate to these types of projects, and I've ultimately come to the conclusion that maintaining any service with users sucks. They can be the best users you could ask for (and in this case, are), but they're still users. They have expectations, and they have problems, and they have questions, and they have feature requests, and so on.

When I sit down after a long day of work, I want to work on something that I want to work on then and there, not something that I feel obligated to work on because a user is depending on it being available. I want to work on things that I can break willy-nilly and no one will even notice.

"Fixing" verify in the Star Citizen launcher

2024-10-18T00:00:00-04:00

I've been "playing" Star Citizen (SC) for a few years now entirely on Linux, and it's been a bit of a rollercoaster. It's not officially supported on Linux, but these days it runs pretty well with Wine to the point where I get higher framerate than I do on Windows.

But, of course, something breaks everytime there's an update.

Unfortunately with the new and shiny launcher 2.0.5, the verify functionality on Linux will just spin forever without progressing. Can we either fix this or work around it to get into the game?

When we check the launcher's logs, we see:

{ "t":"2024-10-18 19:01:24.184", "[main][info] ": "[Installer] - Verifying Star Citizen LIVE 3.24.2-live.9381373 at Z:\\mnt\\data\\Games\\star-citizen\\drive_c\\Program Files\\Roberts Space Industries\\StarCitizen"  },
{ "t":"2024-10-18 19:01:24.185", "[main][info] ": "[Installer] - Verifying permissions on Z:\\mnt\\data\\Games\\star-citizen\\drive_c\\Program Files\\Roberts Space Industries\\StarCitizen"  },
{ "t":"2024-10-18 19:01:24.373", "[main][info] ": "[Analytics] event sent : App:Verify:Start"  },
{ "t":"2024-10-18 19:01:25.406", "[main][error]": "[Installer] - Could not fix permissions on directory: Z:\\mnt\\data\\Games\\star-citizen\\drive_c\\Program Files\\Roberts Space Industries\\StarCitizen"  },
{ "t":"2024-10-18 19:02:14.247", "[main][info] ": "[API] - request: https://status.robertsspaceindustries.com/index.json"  },
{ "t":"2024-10-18 19:02:39.712", "[main][info] ": "Removing all remaining login data"  },

Right off the bat, we see that the launcher is trying to fix permissions on the installation directory, and failing. The UI should handle this gracefully, but it doesn't and so we end up spinning forever. We know our permissions are fine, so can we bypass this?

First, lets try to find where this log message is getting generated:

ag -a "Could not fix"
Binary file RSI Launcher/resources/app.asar matches.

Interesting! The launcher is an asar archive, which is a simple tar-like archive format that Electron uses to package its applications. We can extract this archive and look at the source code:

npx @electron/asar extract RSI\ Launcher/resources/app.asar output

Now we can search the extracted source code for the error message:

ag -a "Could not fix" output
output/lib/src/services/installer/installer.js
319: electron_log_1.default.error(`[Installer] - Could not fix permissions on directory: ${gameDirectory}`);

If we take a look at line 319 in output/lib/src/services/installer/installer.js, we see:

if (verify) {
    electron_log_1.default.info(`[Installer] - Verifying permissions on ${gameDirectory}`);
    try {
        await this.fixPermissions(gameDirectory);
    }
    catch (e) {
        electron_log_1.default.error(`[Installer] - Could not fix permissions on directory: ${gameDirectory}`);
        const newError = new Error(`[Installer] - Error while verifying permissions on ${gameDirectory}`);
        newError.name = 'FixPermissionsError';
        newError.stack = `${e}`;
        return this.mainWindow?.webContents.send(app_shared_1.ipcEvents.installer.INSTALLER_INSTALL_FAILED, newError);
    }
}

This tries to call a fixPermission utility, which is actually implemented as a bundled .exe:

/**
 * Attempts fixing the permissions for the game installation on a given directory
 * @param directory The directory of which to fix the permissions
 * @throws {Error}
 */
fixPermissions(directory) {
  const installerSupportExecutable = path_1.default.resolve(this.resourcesPath, this.executableName);
  const cmd = `"${installerSupportExecutable}" --fix-library-permissions "${directory}"`;
  return execPromise(cmd);
}

In theory, if this fails to run the main window should get an INSTALLER_INSTALL_FAILED event, but it doesn't seem to be doing anything with that error. Now, we know our permissions are fine, so we don't actually need any of this! We can just delete the entire if (verify) block and repackage the archive:

npx @electron/asar pack output RSI\ Launcher/resources/app.asar

Now if we restart the launcher, verification just works! The reason this stub .exe is failing is due to Powershell not being installed in our Wine prefix. We could install it, but adding another dependency to manage in our Wine prefix isn't ideal when it's only used here. This is a bit of a hack, but it works for now and we can get back to playing the game.

Parsing URLs in Python

2024-03-15T00:00:00-04:00

URLs

Parsing URLs correctly is surprisingly hard. Who even defines what a "correct" URL is? URLs have evolved drastically since they were originally defined in 1994. The WHATWG has a URL specification that is comprehensive and has helped standardize the behavior of URLs across browsers, but this specification still isn't universal and ambiguities like "how many slashes are you allowed" can definitely get on your nerves:

So browsers accept URLs written with thousands of forward slashes instead of two. That is not a good reason for the spec to say that a URL may legitimately contain a thousand slashes. I’m totally convinced there’s no critical content anywhere using such formatted URLs and no soul will be sad if we’d restricted the number to a single-digit. So we should. And yeah, then browsers should reject URLs using more.

However, if you're creating something new and want to handle URLs, the WHATWG's URL specification probably is the best place to start.

URLs in Python

If you're working in Python, you'd probably start with the built-in urllib module. It's been around forever, but unfortunately it's not compliant with any URL specification, either the much older rfc3978:

RFC 3986 is considered the current standard and any future changes to urlparse module should conform with it. The urlparse module is currently not entirely compliant with this RFC due to defacto scenarios for parsing, and for backward compatibility purposes, some parsing quirks from older RFCs are retained. The testcases in test_urlparse.py provides a good indicator of parsing behavior.

... or the WHATWG URL Parser spec:

The WHATWG URL Parser spec should also be considered. We are not compliant with it either due to existing user code API behavior expectations (Hyrum's Law). It serves as a useful guide when making changes.

Having existed for over 16 years, so many projects depend on the urllib module parsing URLs in exactly the way it does that it's unlikely to ever change in any significant fashion.

Ada

The Ada project is a new (2024) attempt to create a URL parsing library that adheres to the WHATWG URL specification and works really, really fast, parsing 7 URLs for every 1 parsed by cURL. Written in C++, it now has bindings to several other languages, including Python, and has become the URL parsing library used by Node.js as of version 18 to great success:

Since Node.js 18, a new URL parser dependency was added to Node.js - Ada. This addition bumped the Node.js performance when parsing URLs to a new level. Some results could reach up to an improvement of 400%.

The ada-python binding is perfectly functional, and the official binding for the project. However, the ada-python bindings are built on CFFI, an approach that has the binding between C and Python written in Python itself, which loses some of the performance benefits of using Ada in the first place when most of the time is spent just making the function call.

can_ada

Daniel Lemire, one of the developers behind the Ada project asked me to take a look at the ada-python bindings and out of that was born can_ada, a new Python binding that uses pybind11 and template magic to generate the binding code, which is then compiled into a Python extension module. This approach has the potential to be much faster than the ada-python bindings, and indeed when comparing the two bindings, the new can_ada binding is about 2x faster than the ada-python bindings which in turn is about 2x faster than urllib.parse!

Name (time in ms)	Min	Max	Mean
test_can_ada_parse	54.1304 (1.0)	54.6734 (1.0)	54.3699 (1.0)
test_ada_python_parse	107.5653 (1.99)	108.1666 (1.98)	107.7817 (1.98)
test_urllib_parse	251.5167 (4.65)	255.1327 (4.67)	253.2407 (4.66)

At the same time, using the pybind11 approach allows for a very succinct and readable binding definition, coming in at just 60 lines of code and almost a 1:1 with the underlying C++ API.

Binary builds are available now for CPython 3.7 to 3.12, and PyPy 3.7 to 3.9 on many OS's and architectures. Install it with pip or get the source:

pip install can_ada

Example

import can_ada
urlstring = "https://www.GOoglé.com/./path/../path2/"
url = can_ada.parse(urlstring)
# prints www.xn--googl-fsa.com, the correctly parsed domain name according
# to WHATWG
print(url.hostname)
# prints /path2/, which is the correctly parsed pathname according to WHATWG
print(url.pathname)

Compare this to the urllib version, which is not WHATWG compliant:

import urllib.parse
urlstring = "https://www.GOoglé.com/./path/../path2/"
url = urllib.parse.urlparse(urlstring)
# prints www.googlé.com
print(url.hostname)
# prints /./path/../path2/
print(url.path)

Flexible JSON Parsing in Python

2023-08-31T00:00:00-04:00

JSON is easily the most common data serialization format used on the web today. It's relatively (and deceptively) simple, human-readable, and parsable by most languages. Python has a built-in json module that, for most people, is Good Enough™. However, it has some downsides:

It's really slow, even with a C extension.
It's fairly strict. It parses "JSON", not "JSON". It's very common to find JSON documents that aren't strictly valid JSON, but are still parsable by some parsers. VSCode, a popular IDE, has a lax JSON parser that allows its configuration file to contain comments. This is not valid JSON, but it's still useful to be able to parse these files.
To parse any JSON document, we have to load the entire document into Python, even if we only cared about a small part of it. The creation of Python objects is always the slowest part of any Python JSON library.
It'll produce JSON that can't be parsed by most other JSON parsers, especially when dealing with numbers.

On the other hand, it's part of the standard library, so it's always available, and it's very easy to use, so it's probably the right choice for most projects that don't have very strict performance requirements or the need to parse non-standard JSON.

Alternatives

There are quite a few alternative Python JSON libraries that offer performance improvements over the built-in json module. Some of the more popular ones are:

orjson - A fast JSON parser written in Rust.
simdjson - A fast JSON parser written in C++ that uses SIMD instructions to achieve extremely high parsing speeds.
ujson - A fast JSON parser written in C.
rapidjson - A fast JSON parser written in C++.

You'll notice none of these are written in Python. They're all written in lower-level languages with a binding layer to use them from Python. This can make them more annoying to install, as you may require a compiler for the respective language if binary wheels are not available for your combination of platform and Python version. For example simdjson requires a modern C++ compiler, and specific CPU features to get the best performance.

Each of these has tradeoffs and benefits:

orjson can't parse comments, but it has very good all-around performance and has excellent performance for serializing certain Python objects, like UUIDs and Dataclasses.
simdjson is the fastest JSON parser in existence, but it only supports strict parsing and has no support for serialization.
rapidjson supports extensions like NaN/Infinity and parsing comments, but is can be even slower than the standard JSON module in certain cases.
both ujson and rapidjson fail several JSON minefield tests.

A contender

yyjson is an excellent JSON parser written in standard C89 that is strict by default, but can be configured to be more flexible. It's also very fast, competing directly with simdjson and even beating it in many cases without using SIMD instructions. It's compilation size is very small, it supports custom memory allocators out of the box, and it has a mutable API for modifying & creating documents. It passes the JSON minefield tests perfectly.

This checks off just about all the boxes I could hope for. Being standard C89, we can compile it for just about any architecture, any OS, and any set of CPU features made in the last 30 years. It's fast, it's small, it's flexible, and it's correct. Since we can use custom allocators, we can use CPython's memory allocator which will allow users to properly track memory usage using standard Python tools.

So, lets make a Python binding for it.

Introducing the poorly named py_yyjson

py_yyjson is a Python binding for yyjson that makes it easy to use and hides most of the C implementation details. It provides binary wheels for most platforms and has no dependencies, so installation is as simple as pip install yyjson. It's written in C, it's pretty fast, and it's licensed under the MIT license.

Internally yyjson has two different APIs for working with mutable and immutable documents. py_yyjson does its best to hide these details from you using a Document object.

Reading & Writing

We can create a Document in many different ways:

from pathlib import Path
from yyjson import Document

Document({'a': 1, 'b': 2}) # A Python object
Document(b'{"a": 1, "b": 2}') # A binary JSON string
Document('{"a": 1, "b": 2}') # A text JSON string
Document(Path('path/to/file.json')) # A file on disk

We can use ReaderFlags and WriterFlags to control optional behaviour, such as ignoring comments:

from yyjson import Document, ReaderFlags

Document('''{
    // Comments in JSON!?!?
    "a": 1
}''', flags=ReaderFlags.ALLOW_COMMENTS)

We can also allow serializing and parsing Infinity and NaN:

from yyjson import Document, ReaderFlags

Document('{"a": Infinity}', flags=ReaderFlags.ALLOW_INF_AND_NAN)
Document({'a': float('inf')})

We can ignore trailing commas:

from yyjson import Document, ReaderFlags

Document('{"a": 1,}', flags=ReaderFlags.ALLOW_TRAILING_COMMAS)

And of course we can combine any of these options:

from yyjson import Document, ReaderFlags

Document(
  '{"a": Infinity,}',
  flags=ReaderFlags.ALLOW_INF_AND_NAN | ReaderFlags.ALLOW_TRAILING_COMMAS
)

There are several other feature flags, so check them out in the documentation.

JSON Pointers

We can use JSON pointers to access just part of a document. When we do this, we avoid creating Python objects entirely, except for just the part that matches. This can be a huge performance win when you only care about a small part of a large document.

from pathlib import Path
from yyjson import Document

doc = Document(Path("canada.json"))
features = doc.get_pointer("/features")

We can also use JSON pointers to pluck out just part of a document for serialization:

from yyjson import Document

doc = Document({'results': {'count': 3, 'rows': [55, 66, 77]}})
print(doc.dumps(at_pointer='/results/rows'))

This would result in a serialized JSON string containing just [55, 66, 77].

JSON Patch (RFC 6902) and JSON Merge-Patch (RFC 7386)

We aren't limited to just reading part of a document - we can modify just a small part of it as well. For example, we can take our large hypothetical canada.json document and add a new GeoJSON entry to the features list:

from pathlib import Path
from yyjson import Document

doc = Document(Path("canada.json"))
patch = Document([
    {'op': 'add', 'path': '/features/-', 'value': {
        'type': 'Feature',
        'geometry': {
            'type': 'Point',
            'coordinates': [1, 2],
        },
        'properties': {
            'name': 'New Feature',
        },
    }},
])
modified = doc.patch(patch)
print(modified.dumps())

This mutated the document entirely in its lower-level representation without creating any Python objects. If we wanted to use a merge-patch (RFC 7386) we could do that too, for example to replace the entire features list with a single entry:

from pathlib import Path
from yyjson import Document

doc = Document(Path("canada.json"))
patch = Document({
    'features': [{
        'type': 'Feature',
        'geometry': {
            'type': 'Point',
            'coordinates': [1, 2],
        },
        'properties': {
            'name': 'New Feature',
        },
    }],
})
modified = doc.patch(patch, use_merge_patch=True)
print(modified.dumps())

Both the JSON Patch and JSON Merge-Patch implementations are the fastest of any Python library I could find.

Conclusion

py_yyjson isn't the fastest JSON parser in every situation, but it tends to either tie or beat the fastest parsers in most situations while supporting non-standard JSON, it's lightweight with no dependencies, and it's highly portable. Unlike most (all?) other Python JSON libraries, it also allows you to manipulate JSON documents without ever creating Python objects, which can be a huge performance win.

If you're looking for a fast, flexible, and correct JSON parser for Python, give py_yyjson a try. It's brand new and reasonably well tested, but if you manage to break it please open an issue.

Goodbye Jekyll, Hello Pelican.

2023-08-29T00:00:00-04:00

I've been using Jekyll to generate this site on GitHub Pages for many years now. It's been a great tool, but I've been working towards simplifying my technical stack and Jekyll is unfortunately the only reason I have Ruby installed on my system.

I've finally settled on Pelican. It checks off quite a few boxes for me:

It's written in Python, which is a default install on most systems.
It uses Jinja2 for templating, which I already use in many, many projects.
11k stars on GitHub and frequent commits. We won't be stuck maintaining it.
I can use Markdown, or I can reStructuredText, which allows me to re-use content I've written for the sphinx documentation generator.
I can plug in my own Markdown parser, so I can fix some issues that bug me when trying to style the generated HTML.
It's something I haven't used before, and mixing it up sometimes is good for the soul :)

Making a Theme

The old version of this blog used a custom theme I wrote for Jekyll that was fairly minimal. Still, I wasn't happy with it how it ended up looking and a cold page load of the index was a horrifying 76.8kb! Some color contrasts would also be difficult for someone with poor vision to read. So the new theme needs to have meet a couple of requirements:

It needs to be small. The initial page load should be ~10kb.
It needs to be accessible
- Colors are high contrast and the font is legible.
- Screen readers should have no problem navigating the site.
It needs to look decent at all screen widths.
It needs to be simple to maintain, with minimal design and markup.
Should score 100 across the board on Google's Lighthouse audit.

Pelican lets us be very lazy with our theme. I don't write nearly enough to need pagination of the article list, or have separate pages for categories, so we simply...don't write them. We'll create a theme directory with the bare minimum that looks like this:

theme
├── static
│ └── css
│     └── body.css
└── templates
    ├── article.html
    ├── body.html
    ├── header.html
    └── index.html

We create a barebones HTML5 template in body.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>{% block title %}{{ SITENAME }}{% endblock %}</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="{{ SITEURL }}/theme/css/body.css" rel="stylesheet">
</head>
<body>
{% include "header.html" %}
{% block content %}
{% endblock %}
</body>

We create a simple header in header.html:

<header class="header">
  <nav>
    <h1><a href="{{ SITEURL }}/">{{ SITENAME }}</a></h1>
  </nav>
</header>

We create a simple article template in article.html:

{% extends "body.html" %}

{% block content %}
<main>
  <article>
    <header>
      <h1>{{ article.title }}</h1>
    </header>
    <section>
      {{ article.content }}
    </section>
  </article>
</main>
{% endblock %}

And finally an index template in index.html:

{% extends "body.html" %}

{% block content %}
<main>
  <article class="intro">
    <section>
      <p>Hi! I'm Tyler Kennedy. You can find me on
        <a href="https://github.com/tktech">GitHub</a>
        or by <a href="mailto:tk@tkte.ch">email</a>.
      </p>
      <p>Looking for my <a href="{{ SITEURL }}/data/pgp.asc">PGP key?</a></p>
    </section>
  </article>
  <section class="articles">
    {% if articles %}
      <ul>
      {% for article in articles_page.object_list %}
        <li>
          <span class="mono">[ {{ article.date.strftime('%Y-%m-%d') }} ]</span>&nbsp;
          <a href="{{ SITEURL }}/{{ article.url }}">{{ article.title }}</a>
        </li>
      {% endfor %}
      </ul>
    {% endif %}
  </section>
</main>
{% endblock %}

Finally, we tell Pelican to use our theme by adding the following to our pelicanconf.py:

THEME = 'theme'

And that's it, we have our custom theme. So far, it looks like I've managed to tick off every box. The initial page load is at 10.2kb, which includes a CSS reset, quite a bit of un-minified CSS, analytics, and a bunch of CSS rules for syntax highlighting with pygments.

Fixing Links

I don't want to break all of the links to my old articles, so I need to make sure that the new site generates the same URLs. This is fairly easy to do with Pelican, as it has a slug setting that lets us set the URL for each article, combined with changing the global pattern for URL generation. We do this by adding a couple of settings to the pelicanconf.py file:

# Change the URL pattern to match the old Jekyll one.
ARTICLE_URL = 'articles/{date:%Y}/{date:%m}/{date:%d}/{slug}.html'
ARTICLE_SAVE_AS = 'articles/{date:%Y}/{date:%m}/{date:%d}/{slug}.html'

Analytics

We don't want full-blown analytics, but it's always nice to know when someone has linked to an article, so you can participate in any conversation. I've decided to give Plausible a try. It's a minimal (<1kb) and self-host-able (but in this case, we'll use their hosted version) analytics tool that doesn't track any personal information (PI).

Deploying

The preferred method of deploying to GitHub pages seems to have changed since I setup this blog. Instead of deploying the gh-pages branch of your git repo, you can use a GitHub action to build the site and deploy it. We'll start with GitHub's template for static HTML and tweak it a bit:

name: Deploy static content to Pages

on:
  push:
    branches: ["master"]
  workflow_dispatch:

permissions:
  contents: read
  pages: write
  id-token: write

concurrency:
  group: "pages"
  cancel-in-progress: false

jobs:
  deploy:
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Setup Pages
        uses: actions/configure-pages@v3
      - name: Setup python
        uses: actions/setup-python@v2
        with:
          python-version: "3.10"
      - name: Install python requirements
        run: pip install -r requirements.txt
      - name: Build pelican site
        run: pelican content -o output -s publishconf.py
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v2
        with:
          path: 'output'
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v2

We also need to put our build requirements into a requirements.txt at the root of the site:

blinker==1.6.2
docutils==0.20.1
feedgenerator==2.1.0
Jinja2==3.1.2
Markdown==3.4.4
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mdurl==0.1.2
pelican==4.8.0
Pygments==2.16.1
python-dateutil==2.8.2
pytz==2023.3
rich==13.5.2
six==1.16.0
Unidecode==1.3.6

With this, everytime we push to the master branch, GitHub will build the site, and deploy it to GitHub Pages. If you're reading this, it worked!

Running Valheim on Dokku

2023-03-03T00:00:00-05:00

Valheim is a Viking-themed, multiplayer survival game that's a fun game to play with friends. Think Viking-Minecraft and you wouldn't be too far off. Since I'm going skiing this weekend, I figured I'd set up a server for my friends to continue our game while I'm gone.

We can easily host Valheim on Dokku, a Heroku clone (and much more) that you can self-host and is great for hosting all your little projects. We're going to use it to host Valheim in 5 minutes. I'm going to assume you've already got Dokku running, so we won't cover the setup.

Step one, create the application:

dokku apps:create valheim

Step two, mount the volumes to store the game's save files and configuration somewhere they'll persist between updates and reboots:

dokku storage:mount valheim /var/lib/dokku/data/storage/valheim/valheim/server:/home/steam/valheim
dokku storage:mount valheim /var/lib/dokku/data/storage/valheim/valheim/saves:/home/steam/.config/unity3d/IronGate/Valheim

Step three, disable the default nginx proxy, since the game uses UDP to communicate and (currently) Dokku's proxy can't handle this:

dokku proxy:disable valheim

Step four, because we're not using Dokku to manage the ports, we need to tell Dokku to turn off our old containers completely when we deploy new versions. If we don't do this, deploying new versions will fail because the ports are already in use:

dokku checks:disable valheim

Step five, manually map our ports to the world. The default port for Valheim is 2456, so this is probably what you want to use. Whatever port you pick, you also need to expose the next two ports.

dokku config:set valheim DOKKU_DOCKERFILE_PORTS="2456"
dokku docker-options:add valheim deploy "-p 2456:2456/udp -p 2457:2457/udp -p 2458:2458/udp"

Step six, set the environment variables. Update these as required:

dokku config:set --no-restart valheim NAME="My Valheim Server
dokku config:set --no-restart valheim WORLD="MyWorld"
dokku config:set --no-restart valheim PASSWORD="LetsPlay"
dokku config:set --no-restart valheim TZ="America/Chicago"
dokku config:set --no-restart valheim PUBLIC="1"
dokku config:set --no-restart valheim AUTO_UPDATE="0"

And finally, deploy the application:

dokku git:from-image valheim mbround18/valheim:latest

And that's it! You now have a working Valheim server running on dokku.

Running Takahē on Dokku

2022-11-26T00:00:00-05:00

Takahē is a new, lightweight, django-based ActivityPub implementation by Andrew Godwin that's best suited for small to medium installs (think you, your family, and your friends) that supports multiple domain names out of the box.

Dokku is a Heroku clone (and much more) that you can self-host and is great for hosting all your little projects. We're going to use it to host Takahē in 5 minutes. I'm going to assume you've already got Dokku running, so we won't cover the setup.

Step one, create the application:

dokku apps:create takahe

Step two, create a postgres database to go with it:

dokku postgres:create takahe

Step three, link our newly created database to our application:

dokku postgres:link takahe takahe

Step four, a small workaround as Takahē uses the environment variable DATABASE_SERVER instead of the Heroku/Dokku default of DATABASE_URL:

dokku config:set takahe TAKAHE_DATABASE_SERVER=$(dokku config:get takahe DATABASE_URL)

Step five, set our other environment variables. Update these as required:

dokku config:set takahe TAKAHE_ENVIRONMENT=production
dokku config:set takahe TAKAHE_SECRET_KEY=ChangeThisToALongRandomString
dokku config:set takahe TAKAHE_STATOR_KEY=ChangeThisToALongRandomString
dokku config:set takahe TAKAHE_MAIN_DOMAIN=f.tkte.ch
dokku config:set takahe TAKAHE_USE_PROXY_HEADERS=true
dokku config:set takahe TAKAHE_CSRF_HOSTS="[\"https://f.tkte.ch\"]"
dokku config:set takahe TAKAHE_ALLOWED_HOSTS="[\"f.tkte.ch\"]"
dokku config:set takahe TAKAHE_CORS_HOSTS="[\"https://f.tkte.ch\"]"

Step six, set the domain name you're going to use for your Takahē install:

dokku domains:set takahe f.tkte.ch

Step seven, lets set up SSL with letsencrypt. Keep in mind you'll need your domain already setup to point to dokku:

dokku config:set takahe DOKKU_LETSENCRYPT_EMAIL=tk@tkte.ch
dokku letsencrypt:enable takahe

Step eight, checkout the code and deploy. Don't forget to :

git clone https://github.com/jointakahe/takahe.git
cd takahe
git remote add dokku <your dokku username>@<your dokku server>:takahe
git push dokku main:master

And finally, lets run the database migrations to populate our database, start a background worker, and create our admin user:

dokku run takahe python manage.py migrate
dokku ps:scale takahe worker=1
dokku run takahe ptyhon manage.py createsuperuser

And you're done!

Tips

Can't upload large images

By default, the nginx template dokku uses will only allow 1mb uploads. Fix that by doing:

echo 'client_max_body_size 50m;' > /home/dokku/takahe/nginx.conf.d/upload.conf
chown dokku:dokku /home/dokku/takahe/nginx.conf.d/upload.conf
service nginx.load

Fixing kdeconnect-sms

2021-09-06T00:00:00-04:00

I received a Google Drive link by SMS with a few family photos. Normally, I would use KDE's handy phone integration, KDE Connect, to open the message on my desktop. Unfortunately, clicking on this conversation - and only this conversation, would cause kdeconnect-sms to crash. Poof. We could just get the link some other way, but the fun of open source is being able to just fix the things that bug you.

What's happening?

As a user all we saw was the application closing. We get no notification, nothing in KDE's error reporting daemon, nada. So lets try the simple thing first and open up kdeconnect-sms in a terminal to see if we get any useful output:

❯ kdeconnect-sms
qrc:/qml/ConversationList.qml:51: TypeError: Cannot read property 'text' of null
qrc:/qml/ConversationList.qml:267: TypeError: Cannot read property 'text' of null
file:///usr/lib/qt/qml/org/kde/kirigami.2/templates/InlineMessage.qml:261:9: QML ActionToolBar: Binding loop detected for property "atBottom"
ktp-people: Account manager ready
qrc:/qml/ChatMessage.qml:109: TypeError: Cannot read property 'length' of undefined
qrc:/qml/SendingArea.qml:69:17: QML TextArea: Possible anchor loop detected on fill.
zsh: segmentation fault (core dumped)  kdeconnect-sms

Ahah, it's a segmentation fault! Since I'm using Arch Linux, which uses systemd, we can use coredumpctl to inspect the core dump:

❯ coredumpctl debug -1
...

The -1 tells coredumpctl to open the last core dump recorded, and debug tells coredumpctl to open it in a debugger which in this case will be gdb. Once it's loaded we can use bt to display a backtrace:

[Current thread is 1 (Thread 0x7efbfe5ffcc0 (LWP 66022))]
(gdb) bt
#0  0x000055a24e3abf95 in ?? ()
#1  0x000055a24e3acd65 in ?? ()
#2  0x00007fe9cfc467a0 in ?? () from /usr/lib/libQt5Core.so.5
#3  0x00007fe9d1874121 in ?? () from /usr/lib/libkdeconnectinterfaces.so.21
#4  0x00007fe9d1874683 in ?? () from /usr/lib/libkdeconnectinterfaces.so.21
#5  0x00007fe9d0c61300 in ?? () from /usr/lib/libQt5DBus.so.5
#6  0x00007fe9cfc3c50f in QObject::event(QEvent*) () from /usr/lib/libQt5Core.so.5
#7  0x00007fe9d06e4d62 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/libQt5Widgets.so.5
#8  0x00007fe9cfc0f3ba in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/libQt5Core.so.5
#9  0x00007fe9cfc124b9 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/libQt5Core.so.5
#10 0x00007fe9cfc689b4 in ?? () from /usr/lib/libQt5Core.so.5
#11 0x00007fe9ce47610c in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
#12 0x00007fe9ce4c9ba9 in ?? () from /usr/lib/libglib-2.0.so.0
#13 0x00007fe9ce473871 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
#14 0x00007fe9cfc67fe6 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#15 0x00007fe9cfc0dd2c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#16 0x00007fe9cfc16294 in QCoreApplication::exec() () from /usr/lib/libQt5Core.so.5
#17 0x000055a24e3a015d in ?? ()
#18 0x00007fe9cf599b25 in __libc_start_main () from /usr/lib/libc.so.6
#19 0x000055a24e3a089e in _start ()

This doesn't give us a lot of information. Since kdeconnect-sms wasn't built for debugging, it doesn't including the debug symbols that would fill in the ?? for us.

Filling in the blanks

When you're trying to debug a binary, having the debugging symbols can be the difference between a 10-minute exercise or a 3-hour expedition through assembly with a debugger. Thankfully, kdeconnect-sms is open source, so we can simply build our own version for debugging. We'll get the source from KDE's gitlab:

❯ git clone https://invent.kde.org/network/kdeconnect-kde.git
❯ cd kdeconnect-kde

There's no mention of it in the README, but you'll also need KDE's ECM, or extra-cmake-modules, in order to build kdeconnect. Since I'm on Arch, this is simply:

❯ sudo pacman -S extra-cmake-modules

And if you don't have it yet, get cmake as well:

❯ sudo pacman -S cmake

Now we'll configure our copy of kdeconnect with debugging symbols, as well as changing the installation location, so it doesn't mess with the system-wide kdeconnect:

❯ mkdir build
❯ cd build
❯ cmake -DCMAKE_INSTALL_PREFIX=~/kde-prefix -DCMAKE_BUILD_TYPE=Debug ..

And finally we actually build and install it:

make -j$(nproc) install

The -j$(nproc) tells make to build in parallel with as many threads as possible, which can speed up this initial compile significantly. Unless you've got a beefy machine, go grab a coffee.

Once it's done compiling, you might be tempted to just run it, but you're more likely to get a crash or a blank screen than anything useful. KDE applications are complex beasts, and KDE applications sometimes need a bunch of scaffolding. Thankfully the build step created a file for you, called prefix.sh in the build directory to make this easy. We also want to kill the background daemon from the global kdeconnect install if it's running, since it may be running a very different version than what we're working on:

❯ killall kdeconnectd kdeconnect-sms
❯ source prefix.sh

Now we start our version of the daemon (and put it in the background with &), which may be in lib or lib64, depending on your environment:

❯ ~/kde-prefix/lib64/libexec/kdeconnectd &

And, finally, we start our version of kdeconnect-sms. If you're doing this in a new shell, remember to source prefix.sh before running it:

❯ kdeconnect-sms

Now when kdeconnect-sms crashes, we can go back and run coredumpctl debug -1 again, but this time we get a very useful backtrace:

[Current thread is 1 (Thread 0x7efbfe5ffcc0 (LWP 66022))]
(gdb) bt
#0  0x00007f343ccb1d22 in raise () from /usr/lib/libc.so.6
#1  0x00007f343b6fb3e0 in KCrash::defaultCrashHandler(int) () from /usr/lib/libKF5Crash.so.5
#2  <signal handler called>
#3  0x00007f343ccb1d22 in raise () from /usr/lib/libc.so.6
#4  0x00007f343cc9b862 in abort () from /usr/lib/libc.so.6
#5  0x00007f343d248910 in QMessageLogger::fatal(char const*, ...) const () from /usr/lib/libQt5Core.so.5
#6  0x00007f343d247cf5 in qt_assert(char const*, char const*, int) () from /usr/lib/libQt5Core.so.5
#7  0x0000561602056c94 in QList<ConversationAddress>::first (this=0x7fff6e344410) at /usr/include/qt/QtCore/qlist.h:361
#8  0x00005616020551d1 in ConversationModel::createRowFromMessage (this=0x561603a894e0, message=..., pos=0) at /home/tktech/projects/kdeconnect-kde/smsapp/conversationmodel.cpp:145
#9  0x0000561602055985 in ConversationModel::handleConversationUpdate (this=0x561603a894e0, msg=...) at /home/tktech/projects/kdeconnect-kde/smsapp/conversationmodel.cpp:189
#10 0x0000561602041ac6 in ConversationModel::qt_static_metacall (_o=0x561603a894e0, _c=QMetaObject::InvokeMetaMethod, _id=2, _a=0x7fff6e344690)
    at /home/tktech/projects/kdeconnect-kde/build/smsapp/kdeconnect-sms_autogen/EWIEGA46WW/moc_conversationmodel.cpp:161
#11 0x00007f343d4a87a0 in ?? () from /usr/lib/libQt5Core.so.5
#12 0x00007f343f73baf0 in OrgKdeKdeconnectDeviceConversationsInterface::conversationUpdated (this=0x561603e78640, _t1=...)
    at /home/tktech/projects/kdeconnect-kde/build/interfaces/conversationsinterface.moc:265
#13 0x00007f343f73b2dd in OrgKdeKdeconnectDeviceConversationsInterface::qt_static_metacall (_o=0x561603e78640, _c=QMetaObject::InvokeMetaMethod, _id=4, _a=0x7fff6e344860)
    at /home/tktech/projects/kdeconnect-kde/build/interfaces/conversationsinterface.moc:138
#14 0x00007f343f73b85a in OrgKdeKdeconnectDeviceConversationsInterface::qt_metacall (this=0x561603e78640, _c=QMetaObject::InvokeMetaMethod, _id=4, _a=0x7fff6e344860)
    at /home/tktech/projects/kdeconnect-kde/build/interfaces/conversationsinterface.moc:223
#15 0x00007f343f70ae96 in DeviceConversationsDbusInterface::qt_metacall (this=0x561603e78640, _c=QMetaObject::InvokeMetaMethod, _id=10, _a=0x7fff6e344860)
    at /home/tktech/projects/kdeconnect-kde/build/interfaces/kdeconnectinterfaces_autogen/EWIEGA46WW/moc_dbusinterfaces.cpp:958
#16 0x00007f343e621300 in ?? () from /usr/lib/libQt5DBus.so.5
#17 0x00007f343d49e50f in QObject::event(QEvent*) () from /usr/lib/libQt5Core.so.5
#18 0x00007f343df46d62 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/libQt5Widgets.so.5
#19 0x00007f343d4713ba in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/libQt5Core.so.5
#20 0x00007f343d4744b9 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/libQt5Core.so.5
#21 0x00007f343d4ca9b4 in ?? () from /usr/lib/libQt5Core.so.5
#22 0x00007f343b75610c in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
#23 0x00007f343b7a9ba9 in ?? () from /usr/lib/libglib-2.0.so.0
#24 0x00007f343b753871 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
#25 0x00007f343d4c9fe6 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#26 0x00007f343d46fd2c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#27 0x00007f343d478294 in QCoreApplication::exec() () from /usr/lib/libQt5Core.so.5
#28 0x0000561602047d76 in main (argc=1, argv=0x7fff6e344f28) at /home/tktech/projects/kdeconnect-kde/smsapp/main.cpp:87

So what's the problem?

Our new and improved backtrace gives us a pretty good idea of what's going wrong, even without looking at the source code.

When trying to create a row from a message...

#8  0x00005616020551d1 in ConversationModel::createRowFromMessage

... it tries to get the first address in a list of addresses...

#7  0x0000561602056c94 in QList<ConversationAddress>::first

... which causes an assert...

#6  0x00007f343d247cf5 in qt_assert(char const*, char const*, int) ()

... which intentially kills kdeconnect-sms.

If we look at the documentation for QList::first() it's pretty clear that it causes an assert if the list is empty:

Returns a reference to the first item in the list. The list must not be empty. If the list can be empty, call isEmpty() before calling this function.

Our backtrace tells us the filename and the line where QList::first() is being called, in this case smsapp/conversationmodel.cpp on line 145:

ConversationAddress sender = message.addresses().first();
QString senderName = message.isIncoming() ? SmsHelper::getTitleForAddresses({sender}) : QString();
QString displayBody = message.body();

There may be a deeper problem here, where the list of addresses on a message should never be blank, but for now we just want to make it more resilient. The sender is only ever used to populate senderName, which looks like it can clearly handle being a blank string. So we'll add an isEmpty() guard around the call to QList::first():

QList<ConversationAddress> addresses = message.addresses();
QString senderName = QString();
if (message.isIncoming() && !addresses.isEmpty()) {
    senderName = SmsHelper::getTitleForAddresses({addresses.first()});
}
QString displayBody = message.body();

Then we rebuild our changes:

❯ make -j$(nproc) install

And after trying to open the SMS conversation that started all of this...it works!

Contributing the fix

Now that we've fixed our segfault, it would be nice if future versions of kdeconnect-sms would include these changes, so we didn't have to patch it every time we wanted to update. For that, we'll contribute our changes "upstream".

The KHeadache

A slight tangent & rant here. KDE uses a central identity service, KDE Identity. You need an account on this to make pull requests on KDE's gitlab, to comment on the forums, pretty much anything in KDE-land.

I created an account on KDE Identity in 2012. I don't remember my username. I'm not entirely sure if you could even specify a username back then, I think it was automatically created from your first and last name. I could not sign up for a new account with my regular email address, as it already existed. I cannot recover my password, as KDE Identity requires the email and the username to recover your account, and offers no way to recover your username. Clicking on the link in "Maintained by KDE Sysadmin" takes you to a bug board page that...requires a working KDE Identity account. This feels like the intro to a joke about the chicken and the egg.

Try typing "KDE identity recover username" into Google. You're going to get results like:

"Password recovery procedure is nearly impossible" - 2017
"kde identify password recovery is idiotic" - 2014
and a lot more.

All of these are rants on the inability to recover your username, or even requiring it in the first place when only an email should be required.

Apparently the "solution" is to email sysadmin@kde.org, but this address isn't actually mentioned anywhere on KDE Identity.

Things like this are a huge headache and a barrier to getting new contributors. I just about stopped right here. Alright, end of rant.

So anyways...

Our change is pretty simple, so we could probably get away with just posting it on the KDE Connect mailing list. It boils down to a short patch:

diff --git a/smsapp/conversationmodel.cpp b/smsapp/conversationmodel.cpp
index 7cd80abc..755f3796 100644
--- a/smsapp/conversationmodel.cpp
+++ b/smsapp/conversationmodel.cpp
@@ -142,8 +142,11 @@ void ConversationModel::createRowFromMessage(const ConversationMessage& message,
         return;
     }

-    ConversationAddress sender = message.addresses().first();
-    QString senderName = message.isIncoming() ? SmsHelper::getTitleForAddresses({sender}) : QString();
+    QList<ConversationAddress> addresses = message.addresses();
+    QString senderName = QString();
+    if (message.isIncoming() && !addresses.isEmpty()) {
+        senderName = SmsHelper::getTitleForAddresses({addresses.first()});
+    }
     QString displayBody = message.body();

     auto item = new QStandardItem;

EVE Online: Jita to Amarr

2021-08-19T00:00:00-04:00

EVE Online is a fun, and possibly psychotic MMORPG that takes place in an expansive stellar cluster with some truly mind-boggling distances involved. To fully explore this universe you can expect to spend over 9 years of your life.

In a game where logistics and industry make news-worth battles possible, calculating optimal routes between systems and stations becomes an important task.

The EVE Universe from the in-game map.

Taking our first look at the EVE Universe can be a little daunting. A total of 8285 solar systems make up the stellar cluster, although not all of them are player-accessible. A few, known as wormholes or J-space, are not permanently attached to any part of the map and their connections roam around. The remaining systems that are permanently connected to each other make up what is known as k-space, or known-space. 13753 stargates form the permanent connection points between the stars. What if we sketched out EVE's main trade-hub, Jita, and its connections to its neighboring systems?

graph TB
    A["Jita"] --> B["Perimeter"]
    A --> C["Mausari"]
    A --> D["New Caldari"]
    A --> E["Niyabainen"]
    A --> F["Sobaseki"]
    A --> G["Mouvolailer"]

If you've ever done graph theory, this hopefully looks pretty familiar. We can easily represent the EVE map as a graph where each solar system is a vertex (or node), in the graph, and each pair of stargates forms an edge between two vertices.

We're going to use Python to create a graph we can use to answer simple questions like "What's the shortest route to get from Jita to Amarr". Later, we'll make it a bit more complex so we can add new conditions, like finding the safest route. In the actual tool this article is based off of, we pull the systems and stargates from EVE's API, known as the ESI, but to keep things simple you can simply download these two JSON files, which contain all the systems and stargates.

There are several popular libraries for working with graphs in Python, such as networkx and igraph. Networkx is a native python library that trades intuitive usage, flexible types and good error messages for high memory usage and slower speed. igraph is a C library with python bindings that is harder to use and less intuitive, but offers significantly better memory usage and faster lookups. For this article, we want simple, so we're sticking with networkx.

Our first step is to load the data. Remember, our systems are our nodes, and the connections between stargates form our edges:

import json
import networkx as nx

graph = nx.Graph()

with open('systems.json', 'rb') as systems_json:
    systems = json.load(systems_json)

    for system in systems:
        graph.add_node(system['system_id'], name=system['name'])

graph.add_node() takes a unique identifier for the node and any extra attributes we want to save on it, in this case the name of the system. One nice thing we get from networkx is the ability to use our system's IDs as the node ID. In igraph, we get incremental numbers as our node IDs. Now that we have all the systems added, we're going to add the edges:

with open('stargates.json', 'rb') as stargates_json:
    stargates = json.load(stargates_json)

    for stargate in stargates:
        graph.add_edge(
            stargate['system_id'],
            stargate['destination_system_id']
        )

We're actually done here. We can answer our original question of "What's the shortest route to get from Jita to Amarr" with what we've done already:

print(
   nx.shortest_path(
        graph,
        30000142, # The System ID for Jita
        30002187  # The System ID for Amarr
    )
)
[30000142, 30000138, 30000132, 30000134, 30005196, 30005192,
    30004083, 30004078, 30004079, 30004080, 30002282, 30002187]

Those numbers are the systems IDs we'd need to pass through to get from Jita to Amarr in the shortest possible distance, and we got the answer in less than a millisecond. Not very useful on their own, but we could easily look up the names from the systems.json file we loaded earlier.

When we called nx.shortest_path(), networkx used an algorithm called Dijkstra's algorithm to efficiently find the shortest path between Jita (30000142) and Amarr (30002187). It's actually a very simple algorithm, and if you don't want to use networkx you could implement it yourself pretty quickly.

Not all roads are worth taking

Just because a route is the shortest on paper doesn't mean it's the best route to take. If you took the original 11-jump route in a freighter, you'd be dead long before you made it to Amarr.

In EVE, the relative safety of a system is represented as a value known a its "security status", which is a number with 3 ranges:

low	high	name
-1	0	null-sec
0	0.5	low-sec
0.5	1	high-sec

Beware the rounding of numbers here, which is a constant source of pain in EVE. The rules for rounding numbers are different depending on where they're used and occasionally just broken to the point where "CCP_Round giveth and CCP_Round taketh" is a running gag. We're keeping it simple for this article, but it's not really low-sec at 0.5, but 0.45. Usually.

Remember that short 11-jump route we found earlier? You would probably have died 4 jumps in as that system is low-sec, allowing other players to attack you with minimal repercussions.

["Jita", 30000138, 30000132, 30000134, "DEATH", 30005192,
    30004083, 30004078, 30004079, 30004080, 30002282, "Amarr"]

If you never make it to your destination, it's not really the shortest route now, is it? We're going to fix this by adding some more information to our graph. Let's go back to where we loaded up nodes and edges and make a few changes:

import json
import networkx as nx

graph = nx.MultiDiGraph()

with open('systems.json', 'rb') as systems_json:
    systems = {
        system['system_id']: system
        for system in json.load(systems_json)
    }

with open('stargates.json', 'rb') as stargates_json:
    stargates = json.load(stargates_json)


for system_id, system in systems.items():
    graph.add_node(system_id, name=system['name'])


for stargate in stargates:
    security = systems[stargate['destination_system_id']]['security_status']

    graph.add_edge(
        stargate['system_id'],
        stargate['destination_system_id'],
        ls=0 if security < 0.45 else 1,
        hs=0 if security >= 0.45 else 1
    )

Right away you should notice a few things. Our graph is no longer a nx.Graph(), it's now a nx.MultiDiGraph(). This is important! Our original graph didn't have a concept of direction, and didn't allow multiple edges between the same nodes. We've also stored two attributes on each edge, hs (high-security) and ls (low-security), which are weights we'll use later to determine if we should take this edge when looking for safe or unsafe paths.

So why have we changed the type of graph we're making? If we sketched our original graph two connected systems would look something like this:

graph LR
    A["Jita"] --> B["Perimeter"]

Then we add the return paths, turning our graph into a directed graph with multiple edges, or a MultiDiGraph() in networkx:

graph LR
    A["Jita"] --> B["Perimeter"]
    B["Perimeter"] --> C["Mausari"]
    B --> A
    C --> B

Now our graph has all the information we need to pick a much safer route. Let's try our shortest path between Jita and Amarr again:

nx.shortest_path(
    graph,
    30000142, # The System ID for Jita
    30002187, # The System ID for Amarr
    weight='hs' # Use the 'hs' attribute for weights
)
[30000142, 30000144, 30000139, 30002802, 30002801, 30002803, 30002768,
 30002765, 30002764, 30002761, 30004972, 30004970, 30002633, 30002634,
 30002641, 30002681, 30002682, 30002048, 30002049, 30002053, 30002543,
 30002544, 30002568, 30002529, 30002530, 30002507, 30002509, 30000004,
 30000005, 30000002, 30002973, 30002969, 30002974, 30002972, 30002971,
 30002970, 30002963, 30002964, 30002991, 30002994, 30003545, 30003548,
 30003525, 30003523, 30003522, 30002187]

That's quite a bit longer, at 46 jumps! However, we're more likely to actually arrive at our destination by taking a route through safer systems.

Note that we've simply done safe or unsafe here. Instead of a 0 or a 1, we could have scaled the weight by how safe the system was. We could also mix in other values, such as "Has someone died here recently?"

Sometimes you want to stay in the less-secure systems. If you're a criminal, you don't want to run into the police now, do you? We would simply use weight='ls' instead of weight='hs' to get a route that we keep us in less-secure space.

Encoding & Decoding MUTF-8 in Python

2021-01-21T00:00:00-05:00

CPython rejected adding a MUTF-8 parser to the standard library. This is unfortunate since there are many times you might want to use MUTF-8 in Python. You might be trying to:

Read NBT, the file format use by Minecraft
Read a JVM ClassFile
Read a DEX file, the binary format used by Android
Query SAP HANA (although this is really CESU-8)
Any object saved with Java Object Serialization

This has resulted in some significant replication of work, with many projects re-implementing MUTF-8 parsing:

So I've plucked the MUTF-8 decoder/encoder out of my Lawu project as the mutf8 package and added an optional C version. Hopefully, this version will not only be correct, but fast as well. The fallback Python version is a tad bit faster than others, and the C version is significantly faster than existing implementations:

Name	Min (ns)
✨ cmutf8	100.9999
mutf8	1,199.9800
androguard-mutf8	1,799.9846

Using our C version in the popular tool androguard reduced the total time to parse the Facebook.apk by 20%. Due to the sheer number of strings in a DEX or JVM ClassFile, any improvement in decoding can have a large impact.

With precompiled binaries for most platforms, and a pure-python fallback, handling MUTF-8 is now a simple pip install mutf8.

Safari 13, Mobile Keyboards, And The VisualViewport API.

2019-09-23T00:00:00-04:00

Suppose you want to keep a menu present at all times at the bottom of the screen on a mobile device. Usually this is fairly trivial and can be solved with just CSS:

.menu {
    position: fixed;
    bottom: 0;
}

Most mobile browsers will work just fine. When the keyboard or other UI elements become visible on the screen, the layout viewport will be resized, events (resize, sometimes scroll) will trigger, and the page will rerender. Our fixed menu will move up so that it's still sitting on top of the keyboard.

There is one glaring exception to this which is Safari on mobile devices. Prior to iOS 8.2 the layout viewport would resize when the keyboard opened. However Apple developers believed this resize to be a performance concern since the page layout may change on a resize (issue #141832). Their solution was to implement another layer on top of the layout viewport and make the keyboard opening completely invisible to both CSS and JavaScript. Seriously,

I spent more than a day in research, but had to concede defeat: it’s undetectable. Viewport changes, media queries, aspect ratios, other events such as blur, nothing gives any clue that the software keyboard has been opened on Safari.

Instead of the page resizing, the layout viewport slides up to make room for the keyboard. The visual viewport (the part of the layout viewport that is actually visible on the screen) gets updated, but until recently there has been no way to detect this change and react to it. Until...

The Visual Viewport API

The solution to these woes is the Visual Viewport API, an experimental standard that allows you to both query and react to changes in the visual viewport. While it's marked as an experiment on MDN, support for it is actually fairly widespread. Safari/iOS 13 added full support for the Visual Viewport API, which means every major browser but old-Edge and IE11 now support it.

Added support for the Visual Viewport API for adjusting web content to avoid overlays, such as the onscreen keyboard.

Don't worry, the compatibility chart may say it's not supported on Safari but that is because I only just addded it and MDN has not yet updated. It's there, it works, I promise.

We can make our fixed menu work on iOS Safari by using the resize and scroll events provided by the new API on the window.visualViewport object.

Since the CSS solution is so much cleaner and renders smoothly as you scroll, you probably only want to use this solution when you have no other choice.

Fast queries against large JSON documents using pysimdjson

2019-07-12T00:00:00-04:00

A very common task when working with large (and small) JSON files is to extract or "query" a small subset of the document. Imagine you're reading a 1GB catalog of products, but you only really care about the products that are in stock. Materializing JSON as Python objects is a very expensive process, so we want to avoid this as much as possible and push the bulk of the work to a lower layer written in another language such as C/C++.

pysimdjson is a (very young) library that wraps the amazing simdjson JSON parser capable of extremely fast parsing (2GB/s+) using the AVX2 instructions available on most modern x86-64 CPUs.

pysimdjson is normally comparable to ujson and other high-performance Python JSON parsers when reading an entire document, but pysimdjson also builds on top of simdjson to provide a simple API for filtering the document without materializing it into Python objects at all.

Caveats

There's a few things to keep in mind before trying pysimdjson:

simdjson does not yet have a streaming API so memory usage will be quite high compared to a streaming parser, dependant on the size of the JSON document being read.
simdjson is optimized for large documents that can fully take advantage of SIMD instructions. Compared to other parsers like ujson it may tie or even be slower on very small documents.
pysimdjson is not really compatible with the built-in JSON interface. For example you cannot specify an encoding as the underlying simdjson library always assumes UTF8. pysimdjson always works on bytes, not str.

Microbenchmark

As we mentioned in the caveats, simdjson just isn't optimized for very small documents. For an example document (166 bytes) such as:

{
    "hello": "world",
    "list": [
        1, 2, 3, 4, 5, 6, 7, 8, 9,
        1, 2, 3, 4, 5, 6, 7, 8, 9,
        1, 2, 3, 4, 5, 6, 7, 8, 9
    ],
    "xyz": 123
}

We get the times:

parser	avg time (s) over 100 runs
built-in json (py3.7.2)	0.000902
ujson	0.000194
pysimdjson (fully materialized)	0.000379
pysimdjson (no materialization)	0.000193

pysimdjson without materializing the parsed JSON into Python objects only ties with ujson which always fully materializes. However, it is still nearly 3 times faster than the built-in parser.

Lets try a much larger document (3.2mb) with many strings, gsoc-2018. This document is a subset of the Google Summer of Code candidates in 2018 and each record within it looks like:

{
    "1": {
        "@context": "http://schema.org",
        "@type": "SoftwareSourceCode",
        "name": "Instructor Interface for Plagiarism Detection",
        "description": "Plagiarism Detection is among the significant ...",
        "sponsor": {
            "@type": "Organization",
            "name": "Submitty",
            "disambiguatingDescription": "Programming assignment ...",
            "description": "Submitty ...",
            "url": "http://submitty.org",
            "logo": "//lh3.googleusercontent.com/..."
        },
        "author": {
            "@type": "Person",
            "name": "Tushar Gurjar"
        }
    },
    ...
}

Trying our benchmark against this document we get:

parser	avg time (s) over 100 runs
built-in json (py3.7.2)	1.452656
ujson	1.289953
pysimdjson (fully materialized)	0.871341
pysimdjson (no materialization)	0.176018

simdjson shines in this case, taking an average of only 0.176s to parse the entire document. Converting the internal simdjson representation of the parsed document into python objects took far longer (0.701s) than actually parsing the document! Still, in both cases we were significantly faster than the alternatives.

For certain problems we can find a middle ground that's still slower than the raw parsing speed of simdjson, but much faster than any other Python parser. Lets assume we were only parsing this document because we were trying to get a list of all the authors. We don't care about any other part of the document so materializing it all is a huge waste of time. Given the same gsoc-2018.json sample we used above we'll use the query .[].author.name to get a list of all the names that appear in the file. Our code looks like this:

import simdjson
with open('gsoc-2018.json', 'rb') as fin:
    pj = simdjson.ParsedJson(fin)
    authors = pj.items('.[].author.name')

This is significantly faster, taking just 0.315s to both parse the document, iterate over it, and materialize the results of the query as a native Python list.

Reading AirStrike 3D II datafiles.

2017-02-27T00:00:00-05:00

I like to browse stackoverflow while I think over problems, and sometimes that leads to some interesting questions. Recently someone asked for help on unpacking the contents of an "old" (2005) PC game's data files to extract the music.

First, lets do a lazy check to see if the poster's assumption that the file is compressed is correct:

» strings -n 20 pak1.apk
jeep_small_rocket_desert
item_missile_big_heat
...

We can see hundreds of readable strings, so it's highly likely that this is not compressed. Yay.

The original question gives us a small log snippet which is important because it makes figuring out the first bit of the header much simpler:

---- Initializing file system ----

pak1.apk - 733 files
pak2.apk - 45 files
pak4.apk - 1 files
F_Init: 
3 data files found.

This tells us two key things:

These three files combine to form a single virtual filesystem.
We now know how many entries we should be able to find in each file to verify our parser is correct.

Let's take a look at the first 16 bytes of each file:

00000000: 0000 803f 9999 0000 2833 2500 dd02 0000  ...?....(3%.....
00000000: 0000 803f 9999 0000 3807 4d00 2d00 0000  ...?....8.M.-...
00000000: 0000 803f 9999 0000 3c07 0100 0100 0000  ...?....<.......

We can see some 3 pretty obvious patterns in the first 16 bytes. The first 8 bytes are probably a magic number identifying this as an apk file. The next 4 bytes are really, really close to the total length of the file, minus a bit (probably len(file) - len(header) - len(footer)). The next 4 bytes are obvious thanks to the log post from the original question: 733, 45, 1. It's the # of files in each archive.

At this point we're kind of stuck. The blocks following this aren't trivial to figure out. Let's take a look a smallest and simplest of the 3 files, pak4.

00010720: f8e9 0000 0000 0000 0000 5452 5545 5649  ..........TRUEVI
00010730: 5349 4f4e 2d58 4649 4c45 2e00 2cc0 5adf  SION-XFILE..,.Z.

Very testing. Near the end of the file we can see the string TRUEVISION-XFILE. + 0x00. This is the magic number for TGA files, an image format. We know where the footer is, but this doesn't tell us how large the file is only where it ends. Since we know there's only 1 entry in this apk file lets try to brute force it. We'll start at the end of the file and keep moving the start of the file until we manage to open a valid TGA file.

import sys
import warnings
from cStringIO import StringIO

from PIL import Image

FOOTER = b'TRUEVISION-XFILE'


def main():
    in_file = sys.argv[1]

    with open(in_file, 'rb') as fin:
        in_file = fin.read()

    start = in_file.find(FOOTER)
    # Ends with a '.' then 0x00.
    end = start + len(FOOTER) + 2

    count = 0
    greatest_found_idx = None
    while count < len(in_file):
        attempted_tga = StringIO(in_file[end-count:end])

        try:
            with warnings.catch_warnings(record=True) as w:
                i = Image.open(attempted_tga)
                if len(w):
                    continue
        except IOError:
            pass
        else:
            greatest_found_idx = end - count
        finally:
            count += 1

    print(greatest_found_idx)
    i.save('image.png')


if __name__ == '__main__':
    sys.exit(main())

It's definitely not pretty, but let's give it a shot:

» python brute.py pak4.apk
1040

Neat, looks like we've found one of the model textures. We've also figured out where the TGA starts in the apk file, at offset 1040. If we remove the parts of the header we know about (16 bytes) we get exactly 1024 bytes of unknown data at the top. Taking a look at the other pak files we can see a clear change in entropy and structure 1040 bytes into each file, so it's fairly likely to be part of the header.

A second pattern becomes obvious here. The footer given in the 2nd header field is variable length, being larger the more files are stored in the apk.

(len(file) - footer_offset) / file_count = 76

This rule holds true for every apk, so we have a footer that consists of a 76 byte entry for each file in the apk. However, just like that odd 1024 byte block at the start, the entropy is extremely high here and there's no logical structure. Because the size is always consistent, we know there's no compression going on here. So the contents are probably "encrypted". However, again the size of each entry is constant which you won't really see with complex encryption schemes.

It's probably a simple substitution cipher.

Bingo

Let's screw around in python.

# Lets read the entire file in to play with it.
In [32]: f = open('pak4.apk', 'rb').read()
# Read the 1024 byte block in the header we aren't too sure about (but I'm
# starting to have a hunch)
In [33]: s2 = f[16:1040]
# Read the (only) footer entry (remember pak4 only has one file)
In [34]: s4 = f[-76:]

... ~100 or so misguided attempts later ...

In [46]: ''.join([chr(ord(c) ^ ord(s2[i])) for i, c in enumerate(s4)])
Out[46]: 'models\\mapobjects\\elektro\\power_plant.tga\x00...'

Hurray! Everything clicks now. The 1024 block at the end of the header is the XOR cipher used for the file table in the footer. We simply XOR each byte in the footer by the respective byte in the cipher.

Okay, lets combine everything we've learned and make a useful class to handle this all for us:

#!/usr/bin/env python
# -*- utf-8 -*-
import sys
import struct


class PakParser(object):
    #: File prefix.
    MAGIC_NUMBER = b'\x00\x00\x80\x3F\x99\x99\x00\x00'
    TABLE_ENTRY_SIZE = 76

    def __init__(self):
        self.file_count = 0
        self.cipher_table = None
        self.file_table = {}

    def load(self, file_obj):
        if not file_obj.read(len(self.MAGIC_NUMBER)) == self.MAGIC_NUMBER:
            raise IOError('invalid magic number')

        # The start of the file listing table (given from the start of the
        # file) and the number of file entries.
        file_table_offset, self.file_count = struct.unpack(
            '<II',
            file_obj.read(8)
        )

        # The cipher table comes after the header fields and is always exactly
        # 1kb.
        self.cipher_table = file_obj.read(1024)

        # Each entry in the file listing table is 76 bytes.
        file_obj.seek(file_table_offset, 0)

        self.file_table = dict(
            (filename.strip('\x00'), (offset, size))
            for filename, offset, size in (
                self.unpack_table_entry(
                    self.decipher(
                        file_obj.read(self.TABLE_ENTRY_SIZE),
                        i * self.TABLE_ENTRY_SIZE
                    )
                )
                for i in xrange(self.file_count)
            )
        )

    def decipher(self, chunk, offset):
        return ''.join(
            chr(
                ord(c)
                ^
                ord(self.cipher_table[(i + offset) % 1024])
            )
            for i, c in enumerate(chunk)
        )

    def unpack_table_entry(self, entry):
        # Filename (64 bytes)
        # Offset (4 bytes)
        # Size (4 bytes)
        # The unknown 4 bytes seem to always be 0.
        return struct.unpack_from('<64sIII', entry)[:3]

    def extract_file(self, file_obj, file_name):
        offset, size = self.file_table[file_name]
        file_obj.seek(offset, 0)
        return file_obj.read(size)


def main():
    with open(sys.argv[1], 'rb') as fin:
        pak = PakParser()
        pak.load(fin)

        sys.stdout.write(pak.extract_file(fin, sys.argv[2]))

if __name__ == '__main__':
    sys.exit(main())

And here's how you use the example:

python pakparser.py pak2.apk "sounds\\wavegun.wav" > wavegun.wav

Loading, editing and creating JVM ClassFiles with Jawa

2016-04-11T00:00:00-04:00

Jawa is a rainy-day project to support inspecting, modifying, and creating JVM bytecode from Python. It's a successor to an earlier project from 2010 which was used to magically parse new versions of Minecraft and find new network packets, entities, sounds, etc. It did this by looking for patterns in the bytecode and reconstructing higher-level objects based on what it found.

These days, there are popular new tools like Krakatau for producing human-readable output, but this kind of project isn't always the best option when you actually react on the results.

Creation

Jawa can construct brand new ClassFiles from scratch - lets try the classic "Hello World!" example:

#!/usr/bin/env python
# -*- coding: utf8 -*-
"""
An example showing how to create a "Hello World" class from scratch.
"""
from jawa import ClassFile
from jawa.assemble import assemble

if __name__ == '__main__':
    cf = ClassFile.create('HelloWorld')

    main = cf.methods.create('main', '([Ljava/lang/String;)V', code=True)
    main.access_flags.acc_static = True
    main.code.max_locals = 1
    main.code.max_stack = 2

    main.code.assemble(assemble([
        ('getstatic', cf.constants.create_field_ref(
            'java/lang/System',
            'out',
            'Ljava/io/PrintStream;'
        )),
        ('ldc', cf.constants.create_string('Hello World!')),
        ('invokevirtual', cf.constants.create_method_ref(
            'java/io/PrintStream',
            'println',
            '(Ljava/lang/String;)V'
        )),
        ('return',)
    ]))

    with open('HelloWorld.class', 'wb') as fout:
        cf.save(fout)

Now lets give it a try:

» java HelloWorld
Hello World!

Success! Just like that, we've assembled a Class that the JVM will happily run. You can compare this to the Jasmin "Hello World!" example, the defacto standard for JVM assembly syntax. Both examples are equally compact and concise. We accomplish this by using the assemble() helper which provides support for psuedo-assembly (including named labels and branches), generating a stream of Instruction and Operand objects. This is what it would look like without that helper:

from jawa.bytecode import Instruction, Operand, OperandTypes

main.code.assemble([
    Instruction.from_mnemonic('getstatic', [
        Operand(
            OperandTypes.CONSTANT_INDEX,
            cf.constants.create_field_ref(
                'java/lang/System',
                'out',
                'Ljava/io/PrintStream'
            ).index
        )
    ],
    ...
])

This is extremely precise and will always result in bytecode exactly as provided (even when it's wrong), but you would quickly go insane doing this by hand so it's recommended to always use the assemble() helper.

Modification

We can also easily modify existing classes. Lets take our Hello World! example from the last section and turn it into a Hello Mars! example.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys

from jawa import ClassFile
from jawa.assemble import assemble


def main():
    with open(sys.argv[1], 'rb') as fin:
        cf = ClassFile(fin)

        # We aren't doing HelloWorld any more, so lets fix the name of
        # our class!
        cf.this = cf.constants.create_class('HelloMars')

        # We could just modify the "hello world!" string in the constant
        # pool, but where is the fun in that? Instead, we're going to
        # disassemble the main method, find the 'ldc' that loads the string
        # constant to the stack, and change it to point to a new constant.
        main = cf.methods.find_one(name='main')

        new_main = []
        for instruction in main.code.disassemble():
            if instruction.mnemonic == 'ldc':
                # We could build an Instruction and Operand object ourselves,
                # or use the `assemble()` utility to do it for us.
                new_main.extend(
                    assemble((
                        ('ldc', cf.constants.create_string('Hello Mars!')),
                    ))
                )
            else:
                # We only wanted to patch the 'ldc', everything else we want
                # to keep.
                new_main.append(instruction)

        main.code.assemble(new_main)

        with open('HelloMars.class', 'wb') as fout:
            cf.save(fout)


if __name__ == '__main__':
    sys.exit(main())

Lets give our newly modified class a try:

» java HelloMars
Hello Mars!

Success!

More!

Jawa has extensive documentation - give it a try.