服装搭配AI应用

APPLICATION Nov 19, 2024

不再为穿什么而苦恼——Gemini 让准备变得轻而易举！！！

厌倦了茫然地盯着衣柜，不知道该怎么搭配你最喜欢的 T 恤？我们都有过这样的经历！这就是为什么我开发了一款应用程序，让服装搭配不再是猜测。利用 Gemini 先进的生成式AI，你现在可以穿着 T 恤自拍，并立即从自己的衣柜中获得搭配的下装推荐！

1、工作原理

此应用程序利用检索增强生成 (RAG) 模式，将矢量搜索的效率与大型语言模型 (LLM) 的细致理解相结合。

以下是细分：

视觉编码：当你上传自拍照时，应用程序会分析你的 T 恤图像，重点关注颜色、图案、款式和领口等方面。 Gemini 会根据你的喜好，为你准备一份详细的服装搭配建议，以搭配你的上衣。此信息将转换为矢量表示，以数字格式捕捉其本质。
衣橱矢量化：想象一下，你拥有整个衣橱的数字表示！在入门期间，你将上传衣服的图片。该应用程序同样会将这些图片转换为矢量，创建可搜索的风格索引。
AI 驱动的匹配：现在到了有趣的部分！使用矢量搜索，该应用程序可以快速从你的衣橱中找到与你选择的 T 恤矢量紧密匹配的下装选项。这会将选择范围缩小到视觉上最兼容的单品。
LLM 验证和细化：虽然矢量搜索提供了一个强大的起点，但 Gemini 的 LLM 增加了一层类似人类的判断。它会考虑诸如 T 恤的款式、场合、当前时尚趋势甚至你的个人风格偏好（您可以在应用程序中指定）等细微因素，以策划最终建议。

2、Gemini 的力量：上下文及其他

Gemini 的 100 多万个 token 上下文窗口在此改变了游戏规则。它允许 LLM：

了解你的整个衣橱：Gemini 可以分析您的完整风格资料，而不仅仅是关注上传的 T 恤，从而提供更加个性化和连贯的服装建议。
通过 Fine Tuning 考虑你的偏好：你喜欢休闲、正式还是前卫的造型？Gemini 会了解你的品味并相应地调整其建议。我们创建了 Gemini 1.5 Pro 模型的微调版本，因此你不必处理通用样式。
紧跟时尚潮流：通过访问来自 Fine-Tuned 数据和衣橱信息的实时信息，Gemini 可确保你的服装始终紧跟潮流。

3、那么我们是如何做到的呢？

首先在 AlloyDB 中创建你的 WARDROBE仓库。这里有一个详细的codelab，介绍我们如何在 AlloyDB 中为我的数据集执行此操作。

接下来微调 Gemini 模型以适应你的时尚感。在此处阅读有关微调的更多信息，并附上实际操作示例！

构建一个 Web 应用程序，与微调后的模型对话，分析你的上装并推荐合适的下装

为此，我使用了 Java Spring Boot 应用程序和一个简单的 Thymeleaf 前端。控制器与微调后的 Gemini 对话，以分析服装并生成适合的款式建议！使用通用提示调用模型的代码片段：（Java 语言）

try (VertexAI vertexAi = new VertexAI(PROJECT_ID, LOCATION); ) {
              GenerationConfig generationConfig =
                  GenerationConfig.newBuilder()
                      .setMaxOutputTokens(2048)
                      .setTemperature(0.4F)
                      .setTopK(32)
                      .setTopP(1)
                      .build();
                      
      GenerativeModel model = new GenerativeModel("<<MODEL>>", generationConfig, vertexAi);
      String context = 
      "The attached image is an image of a person in the foreground with a specific top outfit. The image may contain other details in the background or may not be containing any outfit in the image. Ignore other background details of the image and only describe the outfit the person in the image is wearing. If the image does not show any top wear or outfit, please respond with a warning message: 'Please snap a picture with a tee shirt!'. If the image contains a teeshirt or a similar top wear outfit, then describe the image as it is without any prefix, you do not need to start with 'a photo of a' or 'a picture of a'. Just describe it. Do not make up description on your own, only describe if there is a top outfit in the picture. Example description text: A white tee shirt with blue floral patterns on it visibly cotton in material. Then for the above description text, follow it up with bottom wear outfit recommendation. Recommend 5 independent bottom wear (STRICTLY should recommend bottom wear items only- pants/skirts/jeans/shorts/trousers/leggings) outfits texts separated by commas, that go stylish and fashionably with the top wear outfit description text. Make sure the bottomwear recommendation is not very broad, but specific in terms of color, style and material. Just return the recommendation part of it starting with the prefix 'STYLE RECOMMENDATION: '. Do not return the original description of the tee shirt or top outfit. So your final response should be in the same structure as the following example: A white tee shirt with blue floral patterns on it visibly cotton in material. STYLE RECOMMENDATION: <<placeholder for your recommendations>>"; 
      Content content = ContentMaker.fromMultiModalData(
       context,
       PartMaker.fromMimeTypeAndData("image/png", readImageFile_bytes(baseline_url))
      );              
       GenerateContentResponse response = model.generateContent(content);
       String responseOutput = ResponseHandler.getText(response);
    }catch(Exception e){
            System.out.println(e);
    }

4、使用基于 RAG 的查询进行匹配和验证

查询以获取已匹配和验证的匹配服装数据：

select id, content, uri, regexp_replace(gemini_validation, '[^a-zA-Z,: ]', '', 'g') gemini_validation from (
SELECT id, content, uri, 
 cast(ARRAY_AGG(LLM_RESPONSE) as text) as gemini_validation
FROM (
 SELECT id, content, uri, 
json_array_elements( google_ml.predict_row( model_id => 'gemini-1.5',
     request_body => CONCAT('{
     "contents": [
       { "role": "user",
         "parts":
            [ { "text": "Read this user search text: ', recommended_text, '. Note that each comma separated phrase in the user search text is an independent outfit recommendation. Check if the following product inventory match is close enough to any one of the recommendations: ', content, ' . Return a ONE-LINE response with 3 values: 1) MATCH: if the 2 contexts are reasonably matching in terms of any of the color or color family specified in the list, approximate style match with any of the styles mentioned in the user search text and fabric match with any of the fabric specifications mentioned in the user search text : This should be a simple YES or NO. Choose NO only if it is completely irrelevant to users search criteria. 2) PERCENTAGE: percentage of match, make sure that this percentage is accurate 3) DIFFERENCE: A clear one-line easy description of the difference between the 2 products. Remember if the user search text says that some attribute should not be there, and the record has it, it should be a NO match."
            } ]
        }
] }'
)::json))-> 'candidates' -> 0 -> 'content' -> 'parts' -> 0 -> 'text'
::text 
AS LLM_RESPONSE
   FROM (
         SELECT id, pdt_desc AS content, uri, ' + search_text + ' as recommended_text 
FROM apparels where sub_category = 'Bottomwear'
 ORDER BY embedding <=> embedding('textembedding-gecko', ' + search_text + ')::vector LIMIT 10) AS xyz ) AS X
group by  id, content, uri
) as final_matches 
where regexp_replace(gemini_validation, '[^a-zA-Z,: ]', '', 'g') like '%MATCH%:%YES%'

最内层的查询表示基于向量搜索的匹配过程，我们直接使用来自 AlloyDB 的文本嵌入模型来查找与 Gemini 推荐的样式选项最接近的匹配。

外部查询只采用最接近的匹配，并要求 Gemini 进行近乎人类的验证，以确定该匹配是否真的适合推荐和风格，同时又不失时尚！

使用简单的 Thymeleaf HTML 前端呈现 Gemini 认可的匹配️：

<div id="imagerecommendationsdiv" th:if="${not #lists.isEmpty(recommendation)}">
      <center>
        <div th:each="recc : ${recommendation}">
          <img th:src="${recc}" alt="Generated Image" />
        </div>
      </center>
    </div>

哦，还有无服务器！！！因此，所有这些功能和操作组件都在 Google Cloud 上和谐地无服务器运行。

我已经将 Cloud Run 用于 Spring Boot — Thymeleaf Web 应用程序和 Cloud Run Functions，它们与基于 AlloyDB 的推荐引擎进行检查。