Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-81878

As a marketing manager I want to automate images tagging with recognition capabilities so I can improve image discovery and categorization

    Details

    • Type: Story
    • Status: Closed
    • Priority: Major
    • Resolution: Completed
    • Affects Version/s: None
    • Fix Version/s: Master
    • Component/s: Doc Auto Tagging
    • Labels:
    • Priority Level:
      Low
    • Sprint:
      August_Collab, 7.2 - S1, September, 7.2 - S2 - PM Review, Col | Sprint 4 | Nov19 - Nov30, Col | Sprint 5 | Dec3 - Dec14, Col | Sprint 6 | Dec17 - Jan18, Col | Sprint 9 | Feb18 - Mar5, Col | Sprint10 | Mar6 - Mar15, Col | Sprint 3 | Nov5 - Nov16

      Description

      Context

      Content marketing relies on tons of user-generated or crawled content (images, documents...). But this content will probably come without any structured meta-data or tags so you can't really take advantage of them. Let's say you are creating a marketing campaign for the next Symposium and this event will take place in a venue close to a beach. You want to use images... with a beach, right? With no meta data, finding an image with a beach within your repository will be a hard task (or impossible task depending on the volume of data and how it is organized). To improve asset discovery you and your team will need to manually annotate all the images in your repository using expensive human labor.

      With auto-tagging, you can assign relevant tags to all these images in an automated fashion improving assets discovery and categorization and in the end... the productivity of your team and company.

      Goal

      The goal of this user story is to implement auto-tagging for images

      Details

      Assets supported: images

      Features to identify in an image (it will depend on the provider [local, external] capabilities):

      • Basic descriptive metadata for concepts and objects.
      • Industry specifics: eg, if your industry is food, the user will receive specific food-related tags.
      • Sentiment: analyze facial attributes to auto-tag images based on emotions like happiness, surprise, and confusion
      • Demographics: tag images of people based on age, ethnicity, and gender
      • Flag: tag questionable or inappropriate images.

      Other features to identify: it will depend on the provider configuration.

      We will provide two auto-tagging modes:

      • On-premise (optional, low priority): based on a pre-trained AI model. This mode targets basic use cases (playground). The customer will need to train the model on their own. The default model is based on imageNet training.
      • External service: advanced feature recognition. We will integrate with external services. We will provide integration OOTB with: Google Vision API and Microsoft Cognitive services.

      Configuration:

      • At portal level we will be able to configure:
        • Enable/Disable auto-tagging
        • Auto-tagging provider selection
        • Provider configuration (API configuration, features to detect, confidence threshold)
        • Max number of auto-tags generated per image (the more tags the more indexing the system will need to do)
      • At instance level we will be able to configure:
        • Enable/Disable auto-tagging (if auto-tagging is disabled at portal level, we won't be able to enable it at this level)
        • Auto-tagging provider selection
        • Provider configuration (API configuration, features to detect, confidence threshold)
        • Max number of auto-tags generated per image (the more tags the more indexing the system will need to do)
      • At site level we will be able to configure:
        • Enable/Disable auto-tagging (if auto-tagging is disabled at portal or instance level, we won't be able to enable it at this level)

      Is there any option to apply auto-tagging to all the images we had before enabling auto-tagging?

      Yes, through the command line.

      Acceptance criteria

      • When a user uploads a supported media asset to a site with auto-tagging enabled the DM system will process the media asset(s) and create tags automatically storing these tags in the asset metadata.
      • Auto-tagging will be applied to images with no tags. If an image has tags, auto-tagging won't be applied.
      • The user should be able to remove all auto-tags.

      UX work needed

      • Remove all tags interaction (when disabling? anytime?)

      What could come next:

      • Documents auto-tagging
      • Assign less relevance to auto-tags
      • Automatic actions based on auto-tags: e.g, if an image is tagged as inappropriate then auto-delete.

       

        Attachments

        1. AssertAutoTag.PNG
          AssertAutoTag.PNG
          91 kB
        2. ATLocalization.PNG
          ATLocalization.PNG
          686 kB
        3. autotag.PNG
          autotag.PNG
          806 kB
        4. Google Cloud Vision.png
          Google Cloud Vision.png
          284 kB
        5. GoogleCloudVisionGeneratedTags.PNG
          GoogleCloudVisionGeneratedTags.PNG
          990 kB
        6. TensorFlow.PNG
          TensorFlow.PNG
          59 kB

          Issue Links

          1.
          Create auto tagging infrastructure LPS-83376 Technical Task Closed Alejandro Tardín  
          2.
          Create Google Cloud Vision Image Tag Provider LPS-83377 Technical Task Closed Brian Chan  
          3.
          Create Microsoft Cognitive Service Image Tag Provider LPS-83378 Technical Task Closed Brian Chan  
          4.
          Create Tensorflow Imagenet Image Tag Provider LPS-83379 Technical Task Closed Brian Chan  
          5.
          Create scoped configurations (instance, site) LPS-83383 Technical Task Closed Brian Chan  
          6.
          Add Javadocs to public API's LPS-84729 Technical Task Closed Brian Chan  
          7.
          Auto-tagging documentation LPS-85054 Technical Documentation Closed Nicholas Gaskill (Inactive)  
          8.
          DMAutoTagging#ConfigureAutoTaggingPerSite LPS-87596 Technical Testing Closed Nikki Prudencio  
          9.
          DMAutoTagging#Smoke LPS-87611 Technical Testing Closed Nikki Prudencio  
          10.
          DMAutoTagging#RemoveAutoTag LPS-88416 Technical Testing Closed Nikki Prudencio  
          11.
          DMAutoTagging#NoAutoTagForManuallyTaggedImage LPS-88897 Technical Testing Closed Nikki Prudencio  
          12.
          DMAutoTagging#AssertAutoTagAfterPublishToLive LPS-89070 Technical Testing Closed Nikki Prudencio  
          13.
          A user can auto tag existing images LPS-89073 Technical Testing Closed Nikki Prudencio  
          14.
          Auto Tagging + Localization LPS-89083 Technical Testing Closed Nikki Prudencio  
          15.
          A user cannot enable auto tagging if it is disabled at the portal level LPS-89193 Technical Testing Closed Nikki Prudencio  
          16.
          A user can configure maximum number of tags generated per image at the instance level LPS-89194 Technical Testing Closed Nikki Prudencio  
          17.
          Maximum number of tags set at the portal level should override instance level settings LPS-89195 Technical Testing Closed Nikki Prudencio  
          18.
          Auto Tagging using Google Cloud Vision API or MS Cognitive LPS-89425 Technical Testing Closed Nikki Prudencio  

            Activity

              People

              Assignee:
              nikki.prudencio Nikki Prudencio
              Reporter:
              antonio.aragues Antonio Aragüés (Inactive)
              Recent user:
              Rubén Heras
              Participants of an Issue:
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Packages

                  Version Package
                  Master