Giuseppe b3dd7f18c4 feat: AI Operations (#2391) ...

Implements an operation-based system to accomplish a number of edit
tasks on a JSX+CSS representation of a selected Webstudio tree.

Every JSX element has the instance id as data-ws-id and therefore the
LLM knows what to edit.

The idea is to create a number of operations (aiOperations)
specs/schemas with Zod describing unique edit tasks, make a
discriminated union, convert them to JSON Schema. We then inject this
information and the current JSX+CSS in the LLM prompt asking the model
to generate an array of operations.

The LLM responds with an array of operations (aiOperations) to fulfill
the user request. We then process these doing the necessary
transformations to convert these to a Webstudio-friendly format
(wsOperations) and send them to the client where we apply them (see
applyOperations).

Some of these operations are going to be "intermediate" operations for
now.

Generating quality user interfaces might be challenging with LLMs.

Having to do so within a generic operations framework like the one
described above makes it even more challenging.

For this reason when it comes to generating user interfaces, the
Operation chain instead offers an operation called
generateTemplatePrompt that provides information about insertion point
in the existing UI and a prompt, possibly enhanced.

Another chain can use these information and a more sophisticated prompt
and model to then generate the user interface and return a
generateInsertTemplate wsOperation. Webstudio does exactly this in a
rest endpoint, using the template-generator chain with gpt4 instead of
gpt-3.5-turbo.

2023-10-06 09:07:59 +02:00

..

build-icons.ts

feat: AI Operations (#2391)

2023-10-06 09:07:59 +02:00