## Description
Detect route started returning json in markdown instead of pure json
`json\n[\"generate-template-prompt\"]\n`
## Steps for reproduction
1. click button
2. expect xyz
## Code Review
- [ ] hi @kof, I need you to do
- conceptual review (architecture, feature-correctness)
- detailed review (read every line)
- test it on preview
## Before requesting a review
- [ ] made a self-review
- [ ] added inline comments where things may be not obvious (the "why",
not "what")
## Before merging
- [ ] tested locally and on preview environment (preview dev login:
5de6)
- [ ] updated [test
cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md)
document
- [ ] added tests
- [ ] if any new env variables are added, added them to `.env` file
Closes https://github.com/webstudio-is/webstudio/issues/3517
## Description
Completion would often start with \`\`\`json instead of the actual json
that we want to parse, so I added a command to not start with \`\`\`json
## Steps for reproduction
1. click button
2. expect xyz
## Code Review
- [ ] hi @kof, I need you to do
- conceptual review (architecture, feature-correctness)
- detailed review (read every line)
- test it on preview
## Before requesting a review
- [ ] made a self-review
- [ ] added inline comments where things may be not obvious (the "why",
not "what")
## Before merging
- [ ] tested locally and on preview environment (preview dev login:
5de6)
- [ ] updated [test
cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md)
document
- [ ] added tests
- [ ] if any new env variables are added, added them to `.env` file
## Description
We allow console.info (btw we should also allow warn/error/info), we
only disallow console.log in the linter because that's what we use for
debugging.
## Steps for reproduction
1. click button
2. expect xyz
## Code Review
- [ ] hi @TrySound , I need you to do
- conceptual review (architecture, feature-correctness)
- detailed review (read every line)
- test it on preview
## Before requesting a review
- [ ] made a self-review
- [ ] added inline comments where things may be not obvious (the "why",
not "what")
## Before merging
- [ ] tested locally and on preview environment (preview dev login:
5de6)
- [ ] updated [test
cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md)
document
- [ ] added tests
- [ ] if any new env variables are added, added them to `.env.example`
and the `builder/env-check.js` if mandatory
## Description
1. What is this PR about (link the issue and add a short description)
## Steps for reproduction
1. click button
2. expect xyz
## Code Review
- [ ] hi @kof, I need you to do
- conceptual review (architecture, feature-correctness)
- detailed review (read every line)
- test it on preview
## Before requesting a review
- [ ] made a self-review
- [ ] added inline comments where things may be not obvious (the "why",
not "what")
## Before merging
- [ ] tested locally and on preview environment (preview dev login:
5de6)
- [ ] updated [test
cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md)
document
- [ ] added tests
- [ ] if any new env variables are added, added them to `.env.example`
and the `builder/env-check.js` if mandatory
Fixes:
- Disable Radix components via feat flag
- classNames should be optional when generating templates
- Prevent unwrapped JSX to fail
- Use shared computeSelectorForInstanceId when inserting templates and deleting instances
- Add extra instruction for GPT to fill required values just in case
## Description
Rate Limiting toasts.
Mic enable toast.
Improved Error messages toasts for debugability reasons.
## Steps for reproduction
1. click button
2. expect xyz
## Code Review
- [ ] hi @kof, I need you to do
- conceptual review (architecture, feature-correctness)
- detailed review (read every line)
- test it on preview
## Before requesting a review
- [ ] made a self-review
- [ ] added inline comments where things may be not obvious (the "why",
not "what")
## Before merging
- [ ] tested locally and on preview environment (preview dev login:
5de6)
- [ ] updated [test
cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md)
document
- [ ] added tests
- [ ] if any new env variables are added, added them to `.env.example`
and the `builder/env-check.js` if mandatory
After detecting a command via /rest/ai/detect we invoke the /rest/ai
endpoint with the command. If the command is not a text generation, we
run the operations chain that supports multiple operations in the same
prompt.
Instead of refactoring the entire operations chain to support
configurable operations, for now in this PR I am just prepending the
user prompt with the `command` name in order to help the AI
disambiguate.
When asking specific values that don't have a Tailwind class, the AI
should generate custom classes using Tailwind's square brackets
notation.
Tweaking the prompts to ask for this
As per title the PR splits command detection and execution in two
endpoints.
We will first make a request to the /rest/ai/detect endpoint which will
return an array with one or multiple detected commands for the prompt.
Once we have that we execute them in parallel.
Rewrote the temporary UI completion handler to do all the above. Error
handling will still have to be polished to account for special cases
like rate limiting etc.
Removes markdown code block delimiters from JSX the parser utility and
adds a new utility to extract code blocks from LLM responses.
There was a bug in the previous implementation where we were parsing the JSX stripped out of code blocks
delimiters but when extracting the props from the source string we were slicing the original string with delimiters.
The AI request helper now takes a `Promise<Response>` coming from
calling fetch() outside of the helper eg. `handleAiRequest(fetch(...))`
In this PR I also fixed the casing of GPT* types
Implements an operation-based system to accomplish a number of edit
tasks on a JSX+CSS representation of a selected Webstudio tree.
Every JSX element has the instance id as data-ws-id and therefore the
LLM knows what to edit.
The idea is to create a number of operations (aiOperations)
specs/schemas with Zod describing unique edit tasks, make a
discriminated union, convert them to JSON Schema. We then inject this
information and the current JSX+CSS in the LLM prompt asking the model
to generate an array of operations.
The LLM responds with an array of operations (aiOperations) to fulfill
the user request. We then process these doing the necessary
transformations to convert these to a Webstudio-friendly format
(wsOperations) and send them to the client where we apply them (see
applyOperations).
Some of these operations are going to be "intermediate" operations for
now.
Generating quality user interfaces might be challenging with LLMs.
Having to do so within a generic operations framework like the one
described above makes it even more challenging.
For this reason when it comes to generating user interfaces, the
Operation chain instead offers an operation called
generateTemplatePrompt that provides information about insertion point
in the existing UI and a prompt, possibly enhanced.
Another chain can use these information and a more sophisticated prompt
and model to then generate the user interface and return a
generateInsertTemplate wsOperation. Webstudio does exactly this in a
rest endpoint, using the template-generator chain with gpt4 instead of
gpt-3.5-turbo.
## Description
Implement AI copywriter functionality behind feature flag. The final UI is not included in this branch.
The feature consists of:
- a `copywriter` chain to generate the copy with AI
- a /rest/ai/copy endpoint to handle generation requests. The response
is streamed as plain text
- the temporary test UI which is a tab in the right sidebar.
For every selected instance it allows to generate copy for it and its
subtree. A `collectTextInstances` utility creates an array of text nodes
objects with the following shape:
```ts
{
instanceId: string;
index: number;
type: "Heading" | "Paragraph" | "Text";
text: string; // current text or empty
}
```
The AI responds with an array of these where the text property contains
the generated text. The frontend parses incrementally and updates a
Webstudio instance by `instanceId`.
The `index` is used to replace the proper text node in the
`instance.children` array. When all the children are text nodes these
are merged into a single text node, otherwise the AI would generate
`children.length` copy. We identify this case by setting the `index` to
`-1`.
This PR introduces the baseline `@webstudio-is/ai` package which
includes:
1. A LLM Abstraction (types) for messages format to make requests to a
model
2. An implementation of the above that uses the OpenAI chat completion
endpoint
3. A common interface for chains. Chains are regular async functions
that can execute an arbitrary number of steps, including calling a LLM
via a model client.
The idea is that a builder API route can use any of these chains to
perform a task that requires AI. For example generate text copy, images,
a theme etc.
Bogdan Chadkin