SCTT — Transformer-based 3D localisation

From visual observations to position refinement

The model projects visual features into tokens and encodes those observations. Candidate embeddings are combined with candidate positions, then decoded against the encoded visual information. A refinement module produces the position estimate.

src = self.feature_projector(feats)
src = self.pos_enc(src)
src_mask = ~mask.bool()

tgt = candidate_embeddings + self.candidate_pos_enc(candidate_positions)

memory = self.encoder(src, src_key_padding_mask=src_mask)
decoded = self.decoder(
    tgt, memory, memory_key_padding_mask=src_mask
)

p0, log_var = self.position_refiner(
    decoded, candidate_positions, candidate_log_vars
)
return {"p0": p0, "log_var": log_var}

def forward(self, feats):
    x = torch.cat([
        feats["desc"],
        feats["uv"],
        feats["ray_dir"],
        feats["score"].unsqueeze(-1)
    ], dim=-1)
    return self.proj(x)

From LAZ map tiles to PyTorch3D point clouds

I built a point-cloud preparation and storage workflow around a static 3D map. The thesis documents PDAL-based preparation of six spatial tiles, local geometric feature annotation, and a custom storage interface built on PyTorch3D. The selected factory excerpt packages points and their features into the project's point-cloud representation.

from pytorch3d.structures import Pointclouds

class GroundTruthPointcloudsFactory:
    @staticmethod
    def build(laz_files) -> GroundTruthPointclouds:
        points, features = laz_files.build()
        pc = Pointclouds(points=points, features=features)
        return GroundTruthPointclouds(pc)

Custom point-cloud tooling built with PyTorch3D. The project-specific LAZ loader and storage wrapper are omitted from this preview.

x = (uv[:, 0] - intr.cx) / intr.fx
y = (uv[:, 1] - intr.cy) / intr.fy
z = torch.ones_like(x)
ray_cam = F.normalize(torch.stack([x, y, z], dim=-1), dim=-1)

Training as a configurable experiment

The training workflow can change which recording rounds contribute data at configured epochs. When the selected rounds change, it rebuilds the dataset and data loader. The surrounding notebook also contains learning-rate scheduling, experiment logging, checkpoint selection, and early-stopping logic.

if epoch in train_cfg.round_curriculum_schedule:
    new_filter = train_cfg.round_curriculum_schedule[epoch]
    if new_filter != current_round_filter:
        current_round_filter = new_filter
        train_cfg.data_cfg.round_filter = current_round_filter
        dataset = train_cfg.data_cfg.make_dataset()
        train_loader = DataLoader(
            dataset,
            batch_size=train_cfg.batch_size,
            shuffle=False,
            collate_fn=dataset.collate_fn,
            num_workers=4,
            pin_memory=True
        )

Training-data selection is controlled by the experiment configuration rather than fixed inside the training loop.

train_loss, loss_parts, train_mae = run_epoch(
    model, train_loader, optimizer, scheduler, loss_fn,
    train_cfg.device, train=True, scaler=scaler
)

wandb.log({
    "epoch": epoch + 1,
    "lr": scheduler.get_last_lr()[0],
    "train/loss": train_loss,
    "train/loss_p0": loss_parts["p0"],
    "train/mae_x": train_mae[0].item(),
    "train/mae_y": train_mae[1].item(),
    "train/mae_z": train_mae[2].item(),
}, step=epoch)

Connecting the model to the wider system

The ROS 2 integration preprocesses the image, retrieves spatial candidates using the available navigation position, and passes the image and orientation into the inference pipeline. When an estimate is returned, it converts the position back to the original coordinate scale and publishes it with the frame timestamp.

masked_img, _ = self.pipeline.preprocess_image(
    img, logger=self.get_logger()
)
candidate_batch = None
if utm_xy is not None:
    x, y = utm_xy
    candidate_batch = self.candidate_manager(x, y)

pred = self.pipeline.run_inference(
    frame={"masked_image": masked_img, "R": R},
    p0_candidates=candidate_batch,
    logger=self.get_logger()
)
if pred is None:
    return
position = self.denormalize_p0(pred["p0"])
self.publish_p0(position, stamp)

The model sits inside an application pipeline: image preparation, spatial retrieval, inference, and position publication.

CREATE TABLE sctt_embedding (
    id INT PRIMARY KEY,
    p0 geometry(PointZ, 25832)
);

CREATE INDEX ON sctt_embedding USING gist (p0);

SELECT id FROM sctt_embedding
ORDER BY p0 <-> ST_SetSRID(ST_MakePoint(x, y), 25832)
LIMIT 8;