We are proud to unleash “Thorny Devil” (2021-06), the latest release of the webLyzard Web intelligence platform. This release history document provides a quick overview of novel features and recent development activities. We invite you to try out these features by accessing the UN Environment Web intelligence dashboard. For a more detailed description of the various data services and visualizations, please refer to the official dashboard documentation.
2021-06 (Thorny Devil)
- Geographic Map. (i) Improved granularity and accuracy of displayed bubbles, including street-level geotags for selected regions; (ii) rendering of multiple bubbles per document, according to extracted geographic entities; (iii) revised configuration menu for specifying the number of displayed bubbles and trajectories; (iv) improved rendering of overlapping bubbles; (v) transparency for non-dominant topics.
- WYSDOM Success Metric. Multiple checkbox selection among all bookmarks, associations and metadata categories. This eliminates the need for a mode switch and the restriction to sentiment and desired vs. undesired associations. Bipolar slider widgets to place a selected time series above or below the axis.
- Word Tree. (i) Extended display to provide a longer list of full-text matches selectable as the root node; (ii) new optimization option to exclude stop words and improve sentence clustering.
- Cluster Map. (i) Improved initial animation, (ii) extended tooltip and label display; (iii) revised placement algorithm to exclude small clusters containing fewer than three nodes.
- Context Filter. The previous “Advanced Search” menu option has been replaced by a gear icon located within the text input box to access the context filter settings.
- Keyword Filter. New menu option to avoid specific keywords and entities in the list of associations and as visualization labels. There are two stop list columns – the list on the left only applies to the current session, while the terms on the right are stored permanently.
- Temporal Animation. Controls to set the duration of an animation, the size of incremental steps and the average window length. Synchronized animation capabilities for the trend chart, geographic map, entity map and tag cloud.
- Document Management. (i) User rating system to assign between one and five stars to a documents, accessible via a corresponding metadata sidebar category; (ii) shopping cart functionality to manually select a collection of documents to be exported; (iii) full-text document preview in the right sidebar.
- Boost and Demote Terms. Accessible via the floating menu of the content area, two new options allow specifying two different term lists to modify the relevance ranking: documents containing a boost/demote term will be shown at the beginning/end of the results list.
- Query Highlighting. (i) Improved highlighting of prefix and postfix matches; (ii) correct processing of query term matches in document titles; (iii) support of Unicode characters and (iv) improved handling of word boundaries.
- Similar-To Queries. Embedded tag cloud, geographic map, keyword graph and story graph based on the similarity of content vis-à-vis a supplied text document.
- Data Export. (i) Export files as email attachments, in addition to interactively downloading them via the Web browser; (ii) option to include the full text of documents in the PDF search results export.
- Query Engine. Update to Elasticsearch 7.13 for improved performance and stability, new indexing and query strategies for geographic data, improved time- and term-based aggregations as well as faster recovery in case of node failure.
- Mobile Dashboard. Complete rewrite of the mobile application using Vue.js 3, with the option to embed the entire application including context filter and tabs to switch between different visualizations.
- Sidebar Visualizations. (i) Placement of settings menus in the upper left corner; (ii) improved placeholders for the initial rendering or in case of missing data to better reflect expected final appearance.
- Multilingual Features. (i) Extend processing pipeline to provide content from Italian sources; (ii) add support of Dutch interface language; (iii) improve user interface localization by switching to gettext utilities.
- Login and Session Control. (i) Revised user interface and add support of external authentication sources, e.g. SAML2 (Active Directory) and OAuth2; (ii) automatic creation of users and synchronization of access rights based on user roles; (iii) keep-alive health checks to the server to detect interruptions of a session or networking issues.
2020-06 (Sagebrush Lizard)
- Temporal Animation. Extension of the geographic map and charting library to show the evolution of the public debate over time – using sliding average aggregations, accessed via new animation controls in the timeline date picker.
- Geographic Map. Pan and zoom operations dynamically update the map, providing more details without cluttering the display at lower zoom levels.
- Scatterplot. Major revision to render different data types (topics or metadata attributes) along both axes and the bubbles, replacing the standard frequency vs. sentiment display of the previous version.
- Trend Chart. (i) New time series “Impact”, based on frequency but considering the reach of the document’s source; (ii) Complementing the running average, the deviation from the average base value over the past months period allows to compare the relative importance of topics with diverging absolute frequency.
- Story Graph. Performance gains, advances in clustering and label post-processing to better align results with the Associations sidebar, thumbnails for all Web sources.
- Cluster Map. Option to select the number of documents shown and an improved rendering algorithm to better adjust the map to windows with different aspect ratios.
- Keyword Graph. More precise user control when dragging individual nodes.
- Layout. Visual appearance adjustments and new highlighting procedures for charts and various visual widgets.
- Metadata Sidebar. New attributes including Emotions (according to Plutchik’s Wheel of Emotions), Recency (based on temporal classification into four quartiles) and Top Sources, leveraging the color-coding capabilities of the Topic Comparison mode.
- Adaptive Tooltip. Horizontal bar chart to show the distribution of selected topics and metadata attributes together with a multi-color line chart to show their development over time.
- Metadata Aggregation. Modified searches and filter operations that give preference to the more accurate sentence-level aggregations over document-level aggregations whenever possible.
- Disambiguation. Phrase editor option to specify how many regular expression terms need to match for inclusion into the set of search results. This can improve the precision of the query at the cost of lower recall, especially for terms that are ambiguous without additional context information.
- Negation Detection. (i) Standard prefixes, grouped by part of speech, to support on-the-fly negation detection; (ii) custom prefix processing for additional user control.
- Entity List. Sorting persons, organizations and locations triggers a remote query to retrieve global results, as compared to just re-sorting the already retrieved list.
- Advanced Search. (i) New field type to allow searching in “Text or Title” without having to use a Boolean OR; (ii) option to search for multiple terms appearing in the same sentence; (iii) validity check for source identifiers.
- Timeline Datepicker. Refactored to allow a more precise date range setting and fix a daylight savings time issue.
- Index Structure. Update of the indexer and advanced search to capture (and query for) the length of a document in terms of contained characters, words and sentences.
- Performance Optimizations. Updating widgets to version 5.x of D3.js and reducing bundle size by removing superfluous CSS definitions improved download speed and browser performance.
WYSDOM Success Metric
- Settings Menu. Dedicated sidebar that opens automatically on activation, containing (i) sliders to set the relative weights of the indicators, (i) actions to reset the weights and show/hide the overall WYSDOM score, and (iii) the selection of desired and undesired topics.
- Temporal Granularity. Use the current chart settings (daily vs. hourly intervals) for WYSDOM queries and visualizations.
- Statistical Indicators. If there are indicators associated with the selected bookmark (e.g. audience metrics such as visitors and page views), these indicators can be accessed and adjusted using separate slider elements.
- Standalone Version for embedding the chart in third-party applications, with a new color scheme to better highlight desired and undesired associations.
- Knowledge Graph. (i) Integration of awareness days, (ii) endpoint and documentation for the entity type “events”; (iii) endpoint to add custom regular expressions for an entity, stored in the triple store, (iv) cleanup of obsolete data, (v) migration of the cache to Elasticsearch 7.x for increased performance.
- Sentiment Analysis. (i) Revision of the unicode emoji lexicon; (ii) sentiment annotation service for Dutch content, initial version that is based on an adjective-only sentiment lexicon.
- Geotagger. Unique street names for cities with a population of 50,000 or more within the DACH region, including alternate names, parent city and coordinates.
- Report Generator. Download dropdown option to select either sentiment or content source for color-coding the results.
- Synonym API. Synonym and antonym suggestions, including retrieval of inflected versions and filtering by part of speech.
2019-06 (Red Iguana)
- Search Results Selector. Redesigned hierarchical layout with five main categories – Documents, Sentences, Sources, Entities and Relations. Typically, each of these categories contains at least one list view and one visualization (e.g. word tree for sentences or scatter plots for entities and sources).
- Info Bar. Context-aware help text feature as part of the portal header, including a drop-down menu to replace the help text with the most frequent sources or references to named entities (persons, organizations, locations).
- Left Sidebar. Look and feel aligned with the visual tools of the right sidebar; support of multiple categories as well as drag and drop operations; revised metadata category, including source country as an additional attribute.
- Source List. Replaced frequency by impact as the default sorting attribute to highlight the most influential sources.
- Threaded View. Support for viewing and navigating hierarchical conversation threads and document-to-document relations.
- Keyword Blacklist. New configuration menu option to specify terms not to be considered in the list of associations and the various keyword-based visualizations (complementing the server-side blacklist module)
- Performance. Reduced initial load time by removing dependencies and separating optional components – e.g. WYSDOM and radar chart.
- Multi-Color Mode. Alternative color scheme based on the selected topics or metadata attributes, visible in the portal mode “Comparison”; including stacked horizontal bar chart to reflect result composition in the source and entity lists.
- Keyword Graph. Pre-fetching required data and calculating the entire graph prior to force-directed rendering leads to a more stable and visually appealing layout in both the dashboard and the PDF version.
- Geographic Map. Support of polygon data, extended dashboard controls, improved vector tiles and refactoring of the entire module.
- Trend Charts. Data granularity (days, minutes, hours) as new floating menu options, including “automated” that adapts to the selected time interval.
- Legends. Implementation of customizable legends for all bubble-based visualizations (geographic map, scatterplot, keyword graph).
- Tag Cloud. Improved tag distribution and zoom behavior.
- Story Graph. Automated sorting to improve the ranking of documents within a given story; increased performance and minor bug fixes.
- Image Server. Added support for favicons in multiple resolutions.
PDF Report Generator
- Knowledge Graph Integration. Provide entity descriptions and show thumbnails for opinion leaders as well as flags for countries (SVG as the preferred format, with PNG fallback).
- 16:9 Slides. Complementing the A4 reports optimized for high-quality print output, this new module generates customizable slide decks in 16:9 format.
- Story Detection. Embedded document links provide direct access to the original content.
- Content Acquisition. Customizable content classification at mirroring time for hybrid Web sources (news, jobs, real estate, etc.); iCal event support and structured forum extraction; GeoJSON filter for CKAN open data integration.
- Sentiment Analysis. Improved n-gram support and named entity processing; better handling of apostrophes and other special characters; preservation of text-based emoticons during the tokenization process; extended set of negation triggers.
- Named Entity Recognition. More granular geotagger based on a new disambiguation algorithm that identifies and confirms unique, non-ambiguous metadata items in the knowledge graph.
- Term Cache. High-performance caching mechanism to speed up internal operations and support third-party applications.
- Reach Metrics. Extended and updated set of metrics (that will become a global dashboard filter); refined normalization across channels.
- API Framework. Provision of new REST APIs (Story API, Keyword API, Event API) for both internal services and third-party applications.
2018-12 (Queensland Lizard)
- Header Design. A set of overlays accessible via the new header replaces various interactions previously distributed across different icons and drop-down menus. The overlays include: Configuration, Data Export, Advanced Search, Date Range and Data Sources.
- Configuration Overlay. Dialog to select (i) the portal mode, which determines whether the total number of references or just co-occurrences with the current search term are used for presenting results, (ii) the sidebar elements and widgets to be shown, and (iii) the interface language.
- WYSDOM Success Metric. In addition to the global default, users can now store desired and undesired settings together with each topic.
- Video Integration. Enhanced compatibility with multiple social media channels, including embedded playback and thumbnail display in the Story View.
- Story Detection. Automated clustering of related documents (= stories) in digital content streams, represented as a streamgraph and corresponding table (including a lead article for each story and a list of related documents).
- Multi-Color Mode. Extension of the geographic map, tag cloud, keyword graph and cluster map modules to support multiple colors for representing bookmarks, associations or metadata elements selected in the left sidebar.
- Geographic Map. Rendering of Scalable Vector Graphics (SVG) tiles instead of bitmap tiles, including better map boundary handling.
- Cluster Map. Revised positioning algorithm to make better use of the available space, especially along the horizontal axis.
Data Services and Infrastructure
- PDF Reporting. Story Detection report including streamgraph and overview of top stories; export of multiple user-selected reports as an integrated PDF file; email alert extension to deliver weekly or monthly reports.
- Social Media Reach. Improved algorithm to compute per-source reach for various social media channels, based on the number of followers and likes.
- Sentiment Analysis. Bigram processing enabled for the negation detection module, avoiding that exceptions such as “not only” trigger a negation.
- Distributed Processing. Addition of three physical nodes to the frontend server cluster and upgrade to Elasticsearch 6.5.1.
2018-06 (Panther Chameleon)
- Opinion Leadership. Scatterplot and corresponding table to show how often opinion leaders were mentioned together with the search term, and whether this happened in a positive or negative context.
- Sentence Analysis. Color-coded word tree to convey major threads in the public debate, followed by tables with recent positive and negative mentions.
- Documentation. Optional appendix page with a summary of the available report types, including a description of their main features.
- Geographic Maps. Custom basemap for a more granular view of the regional distribution of search results, using an updated geographic map module for multiple basemaps.
- WYSDOM Chart. Improved ingestion mechanism for performance and audience metrics; removed dependencies of external libraries beyond D3.js.
- Word Tree. Slider interface element to control the level of detail and number of sentences shown; support of prefix and suffix “*” queries by fetching the most popular matching terms; e.g., eco* -> ecological, economy, etc.
- Incremental Load. Clearer indication of queries in progress by greying out previous datasets and transitioning into the new state once new results become available.
- Advanced Search. To simplify query definition and reduce the number of items, the text editor initially developed for managing regular expressions is now available for additional fields including URL, source identifier, target location and keywords.
- Date Format. Unified display of date references across all dashboard elements using “dd mmm” format.
- Initial Load. Reduced size of initial download including script bundles; more than 200 kb for the mobile dashboard, and 30 kb for the desktop version.
- Content Acquisition. Added support for the ingestion of Reddit content; dynamic mirroring component to collect external resources referenced via outgoing links, e.g. Youtube and Facebook videos.
- Microdata Enrichment. Extraction of Schema.org concepts such as event content and video content from HTML5 microdata, where available.
- Organizational Metadata. Capturing and enriching site owner metadata to improve audience metrics and user-generated content classifications.
2017-12 (Ocelot Gecko)
- WYSDOM Success Metric. Enhanced charting module with a dedicated settings dialog and the option to select desired and undesired topics from any of the defined categories.
- Bar Charts. (1) New horizontal bar chart as an aggregated alternative to the trend chart, particularly useful for less dynamic sources. (2) Vertical bar chart as a more information-rich alternative to the donut chart; (3) redesign of the chart settings dialog to align it with other visualizations.
- Sentiment Line Chart. Increased precision by rendering sentiment values on the sentence level instead of the broader document sentiment.
- Relation Tracker. Support of stacked relations, additional sorting options (entities by type, sources by reach and impact), and improved layout and interaction features.
- Word Tree. To avoid sparse word trees in the case of less frequent terms within a composite query, a revised content retrieval module provides an independent set of sentences for each matching term.
- Continuous Scrolling. For the document and sentence lists, continuous scrolling and dynamic content updates replace the manual paging through the result list in 50-document intervals.
- Initial Load. Improved user experience by immediately displaying placeholder elements and static elements such as chart grids and menus to inform the user about what will be shown in the given module after the data arrived.
- Topic Definition. In addition to creating new topics from scratch, users can now save topic definitions based on the currently active search.
- Wildcard Queries. For queries without term restrictions (“*”), the system hides both the sentence view and the word tree.
- Client Side Error Reporting. Central error tracking service to identify Web browser incompatibilities and client side errors, providing detailed insights for faster software updates and patches.
- Landing Page. Centered search box intended as a simplified landing page for information retrieval-focused applications (as compared to topic comparisons) in order to speed up the initial load and the first query.
- Web Crawling Framework. Completely rewritten, horizontally scalable content acquisition component to shorten crawl intervals, increased recall and improved parsing of the latest Web frontend technologies.
- Document Relations. Increase versatility of the webLyzard document model to support the mapping of threaded dialogs and other dependencies among content items.
- Increased Coverage. Additional Web sources and social media search terms in multiple domains including politics, tourism and consumer brands.
2017-06 (Northern Alligator)
Data Export and Reporting Features
- Report Generator. Automated compilation of on-the-fly PDF summaries, for example to integrate analytic results into weekly management updates; currently supported report types are Trend Analysis, Topic Associations, Cross-Media Analysis, and Geographic Distribution.
- Export Menu. Instead of using a separate sidebar element, restructured and extended export options are now available via the standard header menu.
- E-Mail Alert. Following the removal of the right sidebar, the e-mail alert configuration has been transferred to the topic editor window.
Visual Analytics Dashboard
- Timeline Selector. Complementing the calendar-based date picker, the timeline provides a visual overview of the temporal search results distribution, including interactive elements to to select custom date ranges.
- Brand Reputation Radar. New algorithm to calculate associations that achieves a more balanced normalization across radar chart dimensions.
- Mobile Version. Redesigned landing page with a centrally located search field, optimized loading times and improved overall performance.
- Content Acquisition. Hybrid access method that combines streaming and search APIs to improve recall and enable the ingestion of historic data.
- Named Entity Recognition. Hybrid profiles to dynamically adapt the granularity of the geotagging process and set the minimum size of a city depending on its proximity to a specified region of interest.
The earlier platform releases until March 2017 [A-M] are described in a separate technology archive document.