espressif2022/esp-gsp-src

Readme

# ESP-GSP

English | [中文](README_CN.md)

ESP-GSP (Espressif Graphics Scene Processor) is an ahead-of-time compiled UI
framework for ESP-IDF. It compiles JSON scenes, fonts, and images during the
build, embeds them in firmware, and generates typed C helpers for named UI
elements.

```text
JSON scene -> ESP-IDF build -> generated C API -> ESP-GSP runtime -> display
```

ESP-GSP is intended for embedded products whose screen structure is known at
build time while values, text, visibility, media, and navigation change at
runtime.

## When to use ESP-GSP

ESP-GSP fits control panels, appliance interfaces, dashboards, instruments,
smart-home hubs, and device launchers where the layout is known when firmware
is built. It is especially useful when you want build-time validation, a
generated C API, bounded runtime resources, and one application model across
several display interfaces.

Consider another framework if the application must construct an arbitrary UI
tree at runtime, requires multi-touch, or depends on right-to-left layout or
complex-script shaping.

## How it works

ESP-GSP separates immutable presentation from mutable application state.
Scenes, styles, fonts, local images, and component structure are compiled on
the host. The device keeps only the state and services needed for interaction,
animation, dynamic content, and rendering.

```text
Build time
  JSON scenes + fonts + images
              |
              v
         gspc compiler
              |
              +--> embedded GSP bundle
              +--> generated bundle_gsp.h

Runtime
  input / generated setters / media producers
              |
              v
       queued state updates
              |
              v
    dirty-region command replay
              |
              v
      display presenter -> esp_lcd panel
```

`gsp_add_bundle()` integrates the compiler into the ESP-IDF build. It tracks
scene files and referenced assets, packages compiled scenes and resources into
one embedded bundle, and generates application headers. The runtime validates
the bundle, processes input and state changes, replays only affected drawing
commands, and delegates framebuffer and panel transfer policy to
`esp_display_present`.

There is no device-side retained UI object tree. Named controls and properties
resolve to compiled descriptors and bounded state slots. This keeps normal
application updates small while still supporting runtime text, media, lists,
templates, navigation, and animation.

The responsibility boundary is:

| Part | Responsibility |
|---|---|
| BSP | Initialize the panel, framebuffers, touch, rotation, and byte order |
| Scene | Describe layout, appearance, interaction, and assets |
| Build | Compile and embed scenes through `gsp_add_bundle()` |
| Application | Own product state, handle events, and update named controls |
| ESP-GSP | Validate bundles, route input, update state, render damage, and present frames |

## Feature overview

| Area | Available features |
|---|---|
| Controls | button, label, image, progress, slider, toggle, checkbox, radio, arc, spinner, chart |
| Structure | container, layer, row/column layout, styles, themes, reusable authored components |
| Data and navigation | list, wheel, PageFlow, StackView, Drawer, TabView, Dropdown, Table, Keyboard, Msgbox |
| Graphics | rectangles, rounded rectangles, circles, ellipses, lines, gradients, needles, analog clocks |
| Runtime state | value, checked state, text, visibility, color, bounded geometry, selection, image resources |
| Media | PNG, JPEG, QOI, RLE16, GIF/EAF animation, runtime encoded images, Canvas frames |
| Interaction | single-point touch, taps, long press, value drag, scrolling, flick/settle gestures, scene swipe |
| Motion | property animation, component positioning, page/drawer motion, scene transitions |
| Rendering | RGB565 and RGB888, dirty regions, clipping, alpha, software fallbacks, target acceleration |
| Display paths | RGB, MIPI-DSI, SPI, and QSPI through ESP-LCD presentation strategies |
| Tooling | generated C API, host preview, compiler tests, reference renderer, hardware benchmark |

The complete authoring surface is listed in the
[scene authoring reference](docs/authoring.md).

## Requirements

- ESP-IDF 6.0 or later.
- Python 3.10 or later in the active ESP-IDF environment.
- Pillow installed in that Python environment.
- PyYAML when using the expert `PROFILE` bundle option.
- An application or BSP that initializes an `esp_lcd` panel.

## Install

Add the first public release from an ESP-IDF project:

```sh
idf.py add-dependency "espressif/esp-gsp^0.1.0"
```

Or add it to the application component's `idf_component.yml`:

```yaml
dependencies:
  espressif/esp-gsp: "^0.1.0"
```

For a source checkout, use Component Manager `override_path` or add this
repository to `EXTRA_COMPONENT_DIRS`.

## Run the example

`examples/hello_world` is the smallest complete integration. For the included
ESP32-P4 MIPI-DSI profile:

```sh
cd examples/hello_world
idf.py -D 'SDKCONFIG_DEFAULTS=sdkconfig.defaults;sdkconfig.defaults.esp32p4' \
  set-target esp32p4 build
idf.py flash monitor
```

The example initializes the display, compiles its scene during the build,
starts the UI with optional touch, and updates a named control from application
code. Other checked-in `sdkconfig.defaults.<target>` fragments demonstrate the
same flow; board pins and panel timing must match the actual hardware.

## Quick start

### 1. Describe a scene

Create `ui/main.json`:

```json
{
  "screen": "main",
  "w": 320,
  "h": 240,
  "screen_bg": "#101820",
  "font": "assets/DejaVuSans.ttf",
  "objects": [
    {
      "type": "label",
      "parent": -1,
      "x": 40,
      "y": 48,
      "w": 240,
      "h": 32,
      "text": "Volume",
      "fg_color": "#FFFFFF"
    },
    {
      "type": "slider",
      "parent": -1,
      "name": "volume",
      "x": 40,
      "y": 96,
      "w": 240,
      "h": 32,
      "value": 30,
      "fg_color": "#4CC9F0"
    },
    {
      "type": "button",
      "parent": -1,
      "x": 100,
      "y": 164,
      "w": 120,
      "h": 44,
      "text": "Save",
      "callback": "save"
    }
  ]
}
```

Asset paths are relative to the scene file. Use `name` for controls updated by
the application and `callback` for actions reported to the application.

### 2. Register the bundle

In the application component's `CMakeLists.txt`:

```cmake
idf_component_register(SRCS "app_main.c"
                       PRIV_REQUIRES esp-gsp)

gsp_add_bundle(${COMPONENT_LIB}
    SCENES "../ui/main.json"
    PIXEL_FORMAT rgb565)
```

The build embeds the bundle and generates `bundle_gsp.h`. Scene and asset
changes automatically rebuild it.

### 3. Start and update the UI

```c
#include "esp_gsp_esp_lcd.h"
#include "bundle_gsp.h"

static void on_ui_event(esp_gsp_handle_t ui,
                        const esp_gsp_event_t *event,
                        void *user_ctx)
{
    if (gsp_main_event_is_save(event)) {
        /* Notify an application task. */
    }
}

void app_main(void)
{
    esp_display_present_target_config_t display;
    ESP_ERROR_CHECK(board_display_init(&display));

    esp_gsp_config_t app = gsp_bundle_config();
    esp_gsp_esp_lcd_config_t lcd = ESP_GSP_ESP_LCD_CONFIG_INIT();
    lcd.display = display;
    esp_lcd_touch_handle_t touch = NULL;
    (void)board_touch_init(&touch); /* Touch is optional. */
    lcd.touch = touch;

    esp_gsp_handle_t ui;
    ESP_ERROR_CHECK(esp_gsp_esp_lcd_start(&app, &lcd, &ui));
    ESP_ERROR_CHECK(esp_gsp_on_event(ui, on_ui_event, NULL));
    ESP_ERROR_CHECK(gsp_main_volume_set_value(ui, 60));
}
```

`board_display_init()` and `board_touch_init()` represent BSP functions. See
[`examples/common/hw_init`](examples/common/hw_init) for panel examples.

Generated names follow this form:

```text
gsp_<scene>_<control>_<operation>()
```

The generated header is the authoritative API for a bundle. Generic APIs in
`esp_gsp.h` cover dynamic images, lists, Canvas frames, navigation, and
data-driven integrations. Diagnostics are in `esp_gsp_debug.h`; integration
helpers are in `esp_gsp_advanced.h`.

## Generated API

Include `<symbol>_gsp.h`; the default symbol is `bundle`. It provides the
bundle configuration, scene identifiers, event helpers, template descriptors,
and typed functions for named controls.

Generated functions follow this form:

```text
gsp_<scene>_<control>_<operation>()
```

For `screen: "main"` and `name: "volume"`:

```c
int32_t volume;

ESP_ERROR_CHECK(gsp_main_volume_set_value(ui, 60));
ESP_ERROR_CHECK(gsp_main_volume_get_value(ui, &volume));
ESP_ERROR_CHECK(gsp_main_volume_animate_value_to(
    ui, 80, 250, ESP_GSP_EASE_OUT));
```

The exact functions depend on the control and its dynamic properties. A named
label can generate `set_text()`, an image can generate `set_image()`, a toggle
can generate checked-state helpers, and a clock can generate `set_time()`.
Use editor completion on `bundle_gsp.h` as the authoritative interface for the
current scenes.

Callbacks also generate predicates:

```c
if (gsp_main_event_is_save(event)) {
    /* Apply the corresponding product action. */
}
```

Ordinary applications should prefer these generated helpers. Generic APIs are
available for data-driven integrations. Raw bind, action, object, property,
and template identifiers remain hidden unless
`GSP_BUNDLE_ENABLE_RAW_IDS` is defined before including the generated header.

## Common tasks

### Static images and fonts

Reference assets relative to the scene file:

```json
{
  "type": "image",
  "parent": -1,
  "x": 24,
  "y": 24,
  "w": 64,
  "h": 64,
  "image": "assets/status.png",
  "codec": "auto"
}
```

Set `font` and `default_font_size` on the scene, or override them on a text
object. Referenced assets are rebuilt and embedded automatically. If runtime
text can contain glyphs unknown at build time, add a dynamic font:

```cmake
gsp_add_bundle(${COMPONENT_LIB}
    SCENES "../ui/chat.json"
    PIXEL_FORMAT rgb565
    DYNAMIC_FONT "../assets/NotoSansSC-Regular.otf")
```

### Runtime images, Canvas, and galleries

Use a named image and its generated `set_image()` helper for occasional
encoded images received at runtime. The default call copies the encoded input;
borrowed and ownership-transfer variants are also available.

Use Canvas for continuously produced pixels such as camera previews, video
frames, or live charts. Canvas buffers remain borrowed until their release
callbacks run.

For recycled galleries, declare an image inside a template with
`"dynamic_image": true`. A generated template setter updates a widget
instance; a List row binder can call `esp_gsp_row_set_image()` with the
generated resource slot. Size `config.dynamic_image_slots` for simultaneously
active targets—visible rows plus overscan—not for the total dataset.

See [Application lifecycle](docs/application-lifecycle.md) for COPY, BORROW,
TAKE, callback context, and shutdown ownership rules.

### Lists and runtime data

Lists and wheels keep a bounded set of visible row instances. The application
provides the total item count and a row binder that publishes text, values,
colors, or images for each visible item. Rows are recycled while scrolling;
do not retain a row handle after the binder returns. Row tokens prevent a late
asynchronous image decode from being published into a row already reused for
another item.

Give the authored List a stable name such as `contacts`, then bind application
data through its generated component key:

```c
static const char *s_contacts[] = {"Ada", "Linus", "Margaret"};

static gsp_err_t bind_contact(esp_gsp_handle_t ui, esp_gsp_row_t row,
                              uint32_t item, void *user_ctx)
{
    (void)user_ctx;
    return esp_gsp_row_text(ui, row, s_contacts[item]);
}

esp_gsp_list_t contacts = esp_gsp_list_bind_component(
    ui, GSP_OBJ_KEY_CONTACTS, bind_contact, NULL);
if (contacts == ESP_GSP_LIST_NONE) {
    /* The configured list quota is exhausted. */
} else {
    ESP_ERROR_CHECK(esp_gsp_list_set_total(
        ui, contacts, sizeof(s_contacts) / sizeof(s_contacts[0])));
}
```

For a gallery row template containing a `dynamic_image`, publish encoded image
bytes with `esp_gsp_row_set_image()` and the generated
`GSP_TEMPLATE_<TEMPLATE>_<IMAGE>_RESOURCE_SLOT` constant. Call
`esp_gsp_list_refresh()` after replacing backing data for visible rows.

### Multiple screens and navigation

Register related scenes in one bundle:

```cmake
gsp_add_bundle(${COMPONENT_LIB}
    SCENES "../ui/home.json"
           "../ui/settings.json"
           "../ui/about.json"
    PIXEL_FORMAT rgb565)
```

Navigate with generated scene identifiers, authored actions, or
`esp_gsp_goto_scene()`. PageFlow and TabView move pages within one scene;
StackView provides push/pop navigation; Drawer exposes an edge panel. Control
drag, list scrolling, viewport gestures, taps, and scene swipe share one input
arbitration path so an interactive child takes priority over its container.

See [Navigation and viewports](docs/viewport-transform.md) for the selection
guide and gesture order.

### Shapes, charts, and clocks

Use `shape` for rectangles, rounded rectangles, circles, ellipses, and lines.
Shapes compile into drawing commands and do not create runtime objects. Use
`chart` for authored point data, `needle` for gauges or compasses, and `clock`
for a standard analog face with generated atomic time updates.

```json
{
  "type": "shape",
  "shape": "ellipse",
  "parent": -1,
  "x": 24,
  "y": 24,
  "w": 120,
  "h": 64,
  "bg_color": "#2864B0",
  "border_color": "#D8E8FF",
  "border_width": 2
}
```

## Runtime contract

- Setters are asynchronous on ESP-IDF. Success means the update was accepted.
- Use `esp_gsp_flush()` only for a deterministic test or capture boundary.
- Event, timer, row-binding, and release callbacks run on framework tasks and
  must remain short and non-blocking.
- Stop with `esp_gsp_stop()` from an application task, not from a callback.
- Keep scene dimensions, bundle pixel format, and display configuration
  consistent.
- Keep application buffers in native RGB565 or packed BGR888 layout. Panel
  byte swapping and physical rotation belong in the BSP display target.

See [Application lifecycle](docs/application-lifecycle.md) for synchronization
and image/Canvas ownership details, and
[Display presentation](docs/present_strategy.md) for buffer strategies.

## Display configuration

The scene size, bundle pixel format, and initialized display path must agree.
Provide an accurate `esp_display_present_target_config_t` and normally keep
`ESP_DISPLAY_PRESENT_MODE_AUTO`.

| Display path | Normal strategy |
|---|---|
| RGB / MIPI-DSI | Direct double buffering or partial triple buffering |
| SPI / QSPI with TE | TE-synchronized dirty-area transfer |
| SPI / QSPI without TE | Free-running dirty-area or bounded stripe transfer |

Application-side pixels use little-endian RGB565 or packed B, G, R bytes for
RGB888. Keep runtime image and Canvas data in that native layout. If an SPI or
QSPI panel requires big-endian RGB565 on the wire, set `swap_bytes` in the BSP
display target rather than pre-swapping application buffers. The BSP also owns
physical rotation and framebuffer configuration.

Hardware acceleration is selected only when the operation, target, pixel
format, and memory placement are eligible. CPU paths preserve behavior when an
accelerator cannot be used.

## Bundle options

```cmake
gsp_add_bundle(<component-target>
    SCENES <scene0.json> [scene1.json ...]
    [PIXEL_FORMAT rgb565|rgb888]
    [IMAGE_CACHE_BYTES <bytes>]
    [DYNAMIC_FONT <font.ttf>]
    [SYMBOL <c_identifier>]
    [PROFILE <expert-profile.yaml>])
```

Most applications need only `SCENES` and `PIXEL_FORMAT`. Target acceleration
and the normal presentation mode are selected from ESP-IDF and BSP
capabilities. `PROFILE` loads a YAML expert override and requires PyYAML in the
active ESP-IDF Python environment. Runtime quota and cache defaults are listed
in the [configuration reference](docs/configuration.md).

## Examples and documentation

- [Hello world](https://github.com/espressif/esp-gsp/tree/master/examples/hello_world): minimum ESP-IDF integration.
- [Experience showcase](https://github.com/espressif/esp-gsp/tree/master/examples/showcase): user-controlled product demo for 800x480 and 1024x600 displays.
- [Hardware benchmark](https://github.com/espressif/esp-gsp/tree/master/examples/benchmark): full feature and target exercise.
- [Documentation index](docs/README.md): user and technical references.
- [Host preview](https://github.com/espressif/esp-gsp/blob/master/tools/sim/README.md): simulator usage from a source checkout.

## Limitations

- Scene structure is fixed at build time; runtime variation uses properties,
  templates, lists, dynamic resources, or Canvas frames.
- Touch input is single-point.
- A bundle uses one logical scene resolution and one output pixel format.
- Right-to-left layout and complex-script shaping are not supported.
- Cross-fade requires two scene snapshots; memory-limited targets fall back to
  a direct scene switch.
- Hardware acceleration depends on the target, pixel format, memory location,
  and operation; software fallbacks preserve behavior.

## Troubleshooting

### `gsp_add_bundle requires Python 3.10+ and Pillow`

Activate the intended ESP-IDF environment and install Pillow into that Python:

```sh
. "$IDF_PATH/export.sh"
python -m pip install Pillow
```

### A `PROFILE` build reports that PyYAML is required

Install PyYAML into the same active ESP-IDF Python environment:

```sh
python -m pip install "PyYAML>=6,<7"
```

### The scene does not match the panel

Check the scene `w` and `h`, the panel resolution and orientation, and the
`gsp_add_bundle(PIXEL_FORMAT ...)` value together.

### A generated helper is missing

Give the control a stable `name`, confirm that the property is dynamic for
that control, rebuild, and inspect `bundle_gsp.h`. Add `callback` when an event
predicate is required.

### A setter succeeds but the display has not changed yet

Setters are asynchronous. Use `esp_gsp_flush()` only when the application
needs a deterministic test, screenshot, or synchronization boundary.

### Touch is unresponsive

Confirm that `lcd.touch` is assigned, touch coordinates match the configured
panel orientation, and the target is not hidden or covered by a later object.

## License

Apache-2.0. See [LICENSE](LICENSE).