# PB29 robotic painting arm — public context packet schema: pb29.robot-context.v2 last_updated: 2026-07-12 canonical_url: https://dhvanil.com/pb29.txt human_page: https://dhvanil.com/pb29 repository: https://github.com/DhvanilPatel/PB29 owner: Dhvanil (Nil) location: Berlin, Germany status: active physical mark-making research; supervised A4 pen drawing works This is the one-file briefing for PB29's robotic painting-arm project. Give this URL to an AI agent before asking it to reason about the rig, architecture, calibration, safety, next experiments, or artistic direction. If sections disagree, prefer "Current state" over the historical log. Physical calibration, product specifications, prices, and software APIs can go stale. Verify them before purchases, code changes, or live operation. ## Executive brief PB29 is an observable robotic painting system built around a UFactory xArm 7. It is trying to solve the physical grammar required for a robot to make marks reliably: see the work surface, locate the current tool tip, determine where contact occurs, plan a safe path, execute a useful mark, photograph the result, and preserve the evidence. The project is not simply "an arm that plots SVGs." Its long-term target is brush-based painting under high-level language/vision-model direction. The model may choose an intention or named primitive; deterministic, calibrated code owns motion, contact, paint handling, and safety. The system is a glass box by design. Plans, uncertainty, calibration status, selected primitives, camera evidence, paths, and observed results stay visible. This observability layer is both an engineering discipline and part of the artistic premise: a viewer should be able to see a physical agent perceiving, choosing, and acting. PB29 sits inside Nil's wider painting practice. The name is the Colour Index code for ultramarine. That material origin matters, but this packet is centred on the robotic arm rather than the subject matter of Nil's hand-painted series. ## Priority order When goals compete, use this order: 1. Protect people, the arm, the tool, and the work surface. 2. Establish physical truth with fresh, inspectable evidence. 3. Make a small set of useful mark primitives repeatable. 4. Keep state, uncertainty, and results legible. 5. Add brush and paint handling. 6. Add high-level AI direction only after grounded execution is reliable. Autonomy is not the current bottleneck. The physical truth layer is. ## The target workflow The desired product experience is: 1. Place paper or canvas in the workspace. 2. Install a pen or brush in a compliant mount. 3. Detect the current work surface and derive a safe executable region. 4. Calibrate the actual tip of the current tool. 5. Determine first contact without using a collision stop as the sensor. 6. Choose a named mark-making primitive and its parameters. 7. Ground the primitive into validated robot motion and tool-handling actions. 8. Execute under explicit safety limits. 9. Capture an after-image and compare expected with observed. 10. Save the plan, calibration evidence, path, uncertainty, and result. The current system can complete a supervised subset of this workflow for A4 pen drawing. Fresh arbitrary tools, automatic contact, brushes, and paint remain open work. ## Why this is hard Procedural geometry is the easy part. A robot can be given thousands of elegant curves. The difficult questions are physical: - Which pixels are actually paper rather than tape, mat, glare, or shadow? - Where is that paper in robot-base coordinates? - Where is the current nib relative to the flange? - How does surface height vary across the page? - When did the nib first make contact? - Which inverse-kinematics branch will the real arm use? - Does a path that looks valid in Cartesian space remain safe between samples? - Did the commanded mark land where perception predicted? A few millimetres of stale geometry can break a nib or drive the tool into the surface. A successful vendor SDK return code proves only that a command was accepted; it does not prove that the physical path was safe. ## Current rig As of July 2026: - UFactory xArm 7. - Fixed Elgato Facecam MK.2 overhead camera at 1920 x 1080. - Small wrist camera for close tool and calibration inspection. - Green cutting mat with fiducial markers around an A4 drawing area. - A4 paper taped to the mat. - Edding pen in a lightweight 3D-printed parallelogram flexure/compliant mount. - A4 ChArUco calibration board and wrist-mounted calibration fixtures. - Prusa CORE One+ for mounts, jigs, camera brackets, docks, and prototypes. - Dedicated physical controls for supervised jogging, mode changes, and recovery. - Read-only Next.js operator console exposing calibration trust, camera evidence, robot state, paths, artifacts, and traces. There is no force/torque sensor. Early contact work uses passive compliance, vision, conservative motion, and human supervision. The live robot is local studio equipment. This public packet contains no LAN address, credentials, or permission to move it. ## Visual context These URLs are stable references for people and vision-capable agents. 1. Quiet Attractor — physical result https://dhvanil.com/images/pb29/quiet-attractor-final.jpg The xArm above a completed A4 pen drawing made on 10 July 2026. The study contains 41 paths, 40 pen lifts, and about 8.37 metres of pen travel. 2. Quiet Attractor — registered evidence https://dhvanil.com/images/pb29/quiet-attractor-overlay.jpg The physical result with detected paper boundary, registered axes, and nib clearance overlaid. This is the clearest single image of the glass-box idea. 3. Current overhead rig https://dhvanil.com/images/pb29/rig-overhead.jpg Overhead view of the arm, A4 workspace, cutting mat, fiducials, pen tools, and supervised operator controls. 4. Paper detection evidence https://dhvanil.com/images/pb29/paper-safe-region.jpg Camera image with the detected paper boundary highlighted. The visual mask is evidence; an inset safe region derived from it is executable. 5. Nib detection evidence https://dhvanil.com/images/pb29/nib-detection.jpg Camera view with predicted search geometry and the measured visible pen tip. ROI gating matters because full-frame colour/shape detection produces false positives. 6. Earlier hatch/contact page https://dhvanil.com/images/pb29/hatch-test.jpg A messy intermediate test page. It is not a finished work; it records what the system still misunderstood about contact, path continuity, and the tool. ## Calibration stack The project deliberately separates calibrations that fail for different reasons. ### 1. Fixed-camera intrinsics The lens model maps camera pixels to rays. It becomes stale if camera settings, focus, resolution, or lens geometry change. Current best evidence was solved from an A4 ChArUco capture set: - RMS reprojection error: about 0.520 px. - Resolution: 1920 x 1080. - Status: current for the locked overhead-camera setup. ### 2. Camera-to-robot extrinsics This transform locates the fixed camera in robot-base coordinates. It becomes stale if the camera or robot base moves. Current best wrist-ChArUco hand-eye solve: - 24 input frames, 18 robust inliers. - Mean translation residual: about 0.69 mm. - Median translation residual: about 0.32 mm. - Status: current best, suitable for registration and planning. ### 3. Support surface The mat/support plane relates camera evidence to robot-base Z. Current A4 mat-only fit: - RMS residual: about 1.93 mm. - Maximum residual: about 6.25 mm. - Status: useful for visualization and planning, not authoritative for blind contact. Table captures are diagnostic only in the current cramped workspace. Do not mix far-away table samples into the runtime mat plane merely to increase sample count. ### 4. Paper detection and safe area The canonical paper pipeline uses full-frame segmentation, then fits an A4 quad and derives a physically inset safe region. Important distinction: - Paper mask = visual evidence of the sheet. - Safe draw region = where execution is permitted. Never treat a pretty segmentation overlay as sufficient execution geometry. ### 5. Tool TCP The robot controller reports its flange/default TCP. PB29 separately stores the actual marking tip relative to that flange. The current accepted tool is an Edding pen in a compliant flexure mount. Its TCP is backed by paper and calibration evidence for that exact stack. Any change to pen, nib protrusion, flexure, mount, gripper, keeper, or tool attitude can invalidate it. ### 6. Contact First contact is a measurement problem, not a fixed global Z. Without a force sensor, the intended contact loop is: before image -> one conservative attempt at a fresh XY -> after image -> did a new human-reviewable mark appear? Use a fresh XY for every Z candidate. Once a dot already exists, a later image cannot prove first contact at that location. Collision stops are a final backstop, never the contact detector. ### 7. Mark validation After motion, compare the expected mark region with an after-image. Preserve both successes and failures. A physically executed path is not complete until the system can say what it expected and what it observed. ## Motion policy Near the surface, movement is segmented: vertical lift -> high-clearance lateral travel -> vertical descent Core invariants: - No lateral motion close to the paper. - After contact, error, or uncertainty, recover first with a pure vertical lift at fixed X/Y/orientation. - Use explicit joint-space or branch-locked paths near the workspace. - Validate forward-kinematics samples along a path, not only endpoints. - Save a dry-run trace and replay those exact joint targets for live execution. - Revalidate the saved trace immediately before motion. - Keep local joint steps near the page small and smooth. - Treat roll/pitch/yaw comparisons modulo 360 degrees. - Do not reuse stale low poses after a tool change. The current painting branch and excluded bad branches are explicit project artifacts rather than tribal knowledge. ## Layered architecture ### Layer 0 — physical rig Arm, rigid base, paper/canvas fixture, compliant tool mount, fixed and wrist cameras, lighting, physical controls, and eventually paint wells, rinse, wipe, and tool docks. ### Layer 1 — deterministic execution Coordinate frames, scene limits, path validation, branch policy, lift/travel/ descent choreography, and later fixed-location dip/wipe/rinse actions. ### Layer 2 — mark-making primitives Named, testable routines such as: - stroke - hatch - crosshatch - gradient - glaze - wash - outline - fill - dip - wipe - rinse - dry_wait - photo_check Each primitive should have physical examples, parameter ranges, failure modes, and before/after evidence. ### Layer 3 — sensing and explicit state Calibrated photos, paper/surface evidence, tool identity, calibration freshness, wet/dry assumptions, recent marks, available colours, and uncertainty. ### Layer 4 — grounding Maps an artistic operation into canvas coordinates, paths, tool choice, paint loading, speed, attitude, contact, and validation targets. ### Layer 5 — planner/composer Scripts first; language/vision model later. The planner chooses a high-level operation and parameters. It never emits raw motor motion. ### Layer 6 — learning only where needed Imitation or learned control may eventually help with gestural brushwork, blending, or subtle pressure/speed behaviour. Do not train a model for geometry or fixed choreography that deterministic code already solves. ## Planner versus grounding Correct separation: - Planner: "Make this region cooler and darker with a thin ultramarine glaze." - Grounding: "Use brush B; load PB29 from well 2; wipe twice; follow these calibrated paths at this speed, angle, and contact; then photograph." The first may be model-directed. The second belongs to a constrained executor. ## Current state — 12 July 2026 Demonstrated: - Fixed overhead and wrist-camera calibration suitable for inspection and planning. - Full-frame paper detection plus a separate safe executable region. - A canonical tool-TCP artifact for the current Edding/flexure stack. - Explicit robot-branch policy and guarded motion helpers. - Saved-trace execution for branch-sensitive near-paper paths. - Supervised, registered A4 pen drawings. - Quiet Attractor carried from generated SVG through registration, validated execution, physical result, and visual evidence. - Read-only operator console exposing live state, calibration trust, artifacts, paths, and traces. - Event logging for camera moves, paper changes, tool changes, and flexure changes that invalidate downstream assumptions. Not solved: - Automatic first contact for an arbitrary fresh tool and arbitrary paper. - A surface estimate precise enough to authorize blind contact by itself. - A mature catalogue of useful, repeatable mark primitives. - Brush loading and compliant brush behaviour. - Paint wells, dipping, wiping, rinsing, drying, and contamination control. - Acrylic or oil painting by the arm. - Automatic tool changing. - Force/torque sensing. - Autonomous language/vision-model direction. The honest frontier is calibration and repeatable physical interaction, not path generation or prompt design. ## Milestone log 2026-06 — The physical workshop comes together around an xArm 7, Prusa CORE One+, fixed overhead vision, wrist vision, calibration targets, and early pen mounts. 2026-06-29 — Early contact attempts expose the central safety lesson: vendor Cartesian paths can be physically unsafe near the table even when commands are accepted. Segmented high-clearance motion becomes a project invariant. 2026-06-30 to 2026-07-01 — Paper detection is split into visual evidence and executable safe area. Full-frame segmentation plus deterministic A4 fitting becomes canonical. 2026-07-02 — Camera intrinsics, fixed-camera hand-eye calibration, and the mat support plane are rebuilt from A4 ChArUco evidence, removing a gross support- plane offset. 2026-07-03 — Wrist-camera inspection, explicit robot-branch policy, and saved- trace execution harden the motion stack. Dry-run geometry is no longer assumed to guarantee that a freshly planned live run will choose the same branch. 2026-07-08 — Firmware and redundancy behaviour are rechecked; guarded probe and manual-control workflows are hardened. 2026-07-09 — A lightweight compliant parallelogram pen mount is installed. Tool TCP becomes a canonical, paper-validated artifact for the current Edding/ flexure stack. 2026-07-10 — Quiet Attractor is drawn successfully: 41 paths around a seeded attractor, about 8.37 m of pen travel, with the registered visual overlay saved beside the physical result. 2026-07-12 — The public robotic-arm context packet and image evidence set are published at dhvanil.com. ## Painting materials relevant to the arm The exact studio inventory is public at: https://dhvanil.com/painting-supplies.txt Relevant material anchors: - Acrylics: Schmincke PRIMAcryl, Golden Heavy Body, and related professional colours. - PB29/ultramarine: both a real studio pigment and the project's name. - Oils: Royal Talens Cobra water-mixable oils and medium. - Grounds: white, clear, and black gessos plus specialist primers. - Acrylic additives: retarders, binder, gloss gel, and finishing materials. - Brushes: flats, rounds, filberts, fan, hake, sword liner, dry-brush, stencil, utility, and micro brushes. - Surfaces: archival stretched canvas, smaller boards, and A4 paper for pen and calibration studies. Acrylic is the more plausible first robot paint medium: faster feedback, easier cleanup, and fewer contamination and drying complications than oil. It is still mechanically difficult. Brush compliance, loading, wetness, rinsing, wiping, surface interaction, and cleanup all need explicit choreography. ## Fabrication and workshop context - Prusa CORE One+ with PLA and PETG for fixtures and prototypes. - Heat-set inserts, fasteners, dowel pins, magnets, springs, and abrasives. - Camera and lighting mounts, measuring tools, soldering and wiring equipment. - Current flexure mount is deliberately light and mechanically forgiving. - Printed plastic is a prototype or armature when load or visible material quality matters; critical adapters may need wood, metal, CNC, or commercial hardware. ## Recommended build order from here 1. Make the current pen workflow repeatable from a fresh paper/tool event. 2. Preserve a small physical catalogue of successful and failed pen marks. 3. Implement one useful primitive well, probably a controlled hatch, stroke, or gradient precursor. 4. Improve automatic first-contact evidence without relaxing safety. 5. Install one brush in a passive compliant mount. 6. Use one acrylic colour and a sacrificial surface. 7. Add fixed paint well, wipe, and rinse stations. 8. Track wet/dry assumptions and photograph every primitive. 9. Add more primitives only when existing ones are boringly reliable. 10. Add a language/vision-model planner only after the grounding layer has trustworthy actions to choose from. ## Open questions - What first robot-made primitive is both reliable and artistically useful? - How should brush contact be parameterised without a force sensor? - How should paper and canvas be mounted for repeatable geometry and easy removal? - What evidence should make a calibration "trusted" versus merely plausible? - How should wet paint state be represented and invalidated over time? - How should paint loading be measured: time, path, visual pickup, weight, or a later force signal? - How much of the perception/planning overlay should viewers see live versus in a curated post-hoc trace? - When would a wrist force sensor add enough information to justify mass, cost, calibration, and integration burden? - Which behaviours truly need imitation learning rather than better mechanics and deterministic control? ## Safety and authority for AI agents - This document gives context, not permission to operate hardware. - Read the current repository safety files before proposing or executing live motion. - Run the read-only robot state check before any motion. - Treat camera, tool, flexure, paper, and mount changes as calibration events. - Never use a collision stop as a measurement strategy. - Never treat SDK return code 0 as physical proof. - Never bypass branch checks, workspace limits, saved-trace validation, or high-clearance travel because a path "looks simple." - Do not let an LLM/VLM produce joint commands directly. - After uncertainty, reduce scope: one point, one attempt, one trace, fresh evidence. Current live instructions in the repository override this public summary: - docs/contact-calibration.md - docs/calibration-status.md - docs/camera.md - scripts/01_state.py ## How to work with Nil - Teach from first principles. Explain the physical mechanism, not only the recommendation. - Show reasoning and tradeoffs. - Push back clearly when the difficult 20 percent is the real frontier. - Treat exploratory mulling as legitimate work; do not force closure. - Keep code and systems inspectable: clear over clever, explicit over magical. - Preserve the boundary between high-level artistic direction and low-level grounded execution. - For quick questions, answer quickly. For kinematics, calibration, or hardware tradeoffs, build the intuition carefully. - Nil is both an artist and a systems thinker. Material, visual, mechanical, and software reasoning can coexist in one answer. ## Common agent failure modes - Calling PB29 "an AI plotter" and missing brush media and observability. - Jumping to an autonomous planner before physical primitives are reliable. - Treating a visually convincing overlay as calibration proof. - Collapsing camera, surface, paper, tool, and contact into one magic transform. - Assuming calibration survives a paper, camera, tool, nib, flexure, or mount change. - Optimising procedural complexity rather than physical mark quality. - Recommending force sensing before passive compliance and vision are exhausted. - Repeating stale hardware specifications or prices without verification. - Letting the wider painting-series subject matter dominate a robotic-arm task. ## Useful public links - Human page: https://dhvanil.com/pb29 - This robotic-arm context packet: https://dhvanil.com/pb29.txt - Exact painting inventory: https://dhvanil.com/painting-supplies.txt - Project repository: https://github.com/DhvanilPatel/PB29 - Shareable robot narrative: https://github.com/DhvanilPatel/PB29/blob/main/story.md - Live-rig safety note: https://github.com/DhvanilPatel/PB29/blob/main/docs/contact-calibration.md - Calibration status: https://github.com/DhvanilPatel/PB29/blob/main/docs/calibration-status.md End of PB29 robotic painting-arm context packet.