Never seem to get around posting all the changes that are happening... Most notably, there is now a default new theme in ADAMS using the FlatLAF look and feel, a modern theme for Java Swing. A lot of work went into the imaging modules again, now with more flexibility in terms of how to annotate (polygons: tracing or clicking) and support for custom tools. The new adams-imaging-ext module incorporates deep-learning based tools that aid the user in annotating images (eg using pretrained models or DEXTR).

Fixes

• added support for anti-aliased text in GOE editors and text overlays for improved readability.

• adams-core:

  • updated simple-directory-chooser to 0.0.3 to fix problem with clicking on bookmarks not properly open directory tree.

  • The notification area in the Flow editor no longer splits the error output on ":" but on the top-level actor name, to avoid incorrect splits when colons are part of the actor name.

  • The storage panel of the Breakpoint control panel now updates correctly when the data changes.

  • The Go to menu item in the tree popup menu of the Flow editor now works as expected when there are more than two actor handlers present (going up in the tree).

  • In the Flow editor, a copied single actor (with potential subtree) can now be pasted in the GOE again (the internal representation gets converted to a commandline on demand, which the GOE can parse).

  • The SetTextRenderer for turning Java Set-like data structures into text now starts the numbering at 1 rather than at 2.

  • Validating a flow in the Flow editor now captures more variables.

  • Fixed several NullPointerExceptions relating to property change notifications when using the FlatLAF look and feel.

  • The Parameters tab in the Flow editor no longer shows options that are of type Actor.

  • Added a renderer for Set objects, e.g., used when inspecting storage objects.

• adams-meta: The getActors(Class) method of the AbstractStandaloneGroup class now returns all actors of that class, not just the first one.

• adams-rats-core: The stopExecution method of the Cron input Rat is now synchronized to avoid race conditions in stopping the scheduler.

• adams-redis: The RedisConnection standalone now uses the specified host/port rather than just the default ones.

Changes

• adams-addons-all: added adams-docker as dependency

• adams-annotator: added adams-docker as dependency

• adams-compress: upgraded zip4j to 2.11.3

• adams-core:

  • The ImageHandler in the Preview browser now allows the image reader to be selected by the user.

  • The Preview browser now accepts setups from other content handlers in its options dialog when pasting a setup. It also has menu items for adjusting the window to half (of the screen) width/height.

  • The object inspection tree in the breakpoint view of the Flow editor now uses lazy expansion to speed up display of large (and deeply nested) objects.

  • The GenericObjectEditor now displays log messages in the user interface as well, e.g., when pasting of a setup via the "∨" button (top right corner) goes wrong.

  • upgraded debian-maven-plugin to 1.0.22

  • The ProgressBar sink now has the ability for displaying a button that stops flow.

  • DirectoryValueDefinition and FileValueDefinition now can output absolute paths via the useAbsolutePath option.

  • The Flow editor now has menu items in the window menu for half (of the screen) width/height

  • What options are being displayed in the GenericObjectEditor depends on the current user-mode (Program -> User mode) and what minimum user mode has been defined for an option. E.g., the logging level is only displayed in EXPERT mode now. This allows hiding of options that are not relevant for most users.

  • The DisplayPanelManager sink now allows specifying the location of the divider between entry names and the displayed entry.

  • Upgraded to FlatLAF 3.0 and tweaked JTree parameters to improve appearance.

  • using FlatLAFLight as default look'n'feel now.

  • Submenus in the popup menu of the Flow editor actor tree no longer require a custom class, instead a different definition in the FlowEditor.props file can be used (property: Tree.PopupMenu).

• adams-imaging:

  • The Preview browser handlers ObjectAnnotationsHandler, ObjectCentersFromReport, ObjectLocationsFromReport and ObjectLocationsFromSpreadSheet now allow the user to specify the image reader to use; also use a table rather than a list for displaying the located objects and now offer a search panel; the object panel can be detached into a separate frame as well.

  • The ObjectLocationsHandler in the Preview browser now supports a panel with the object locations.

  • The YoloAnnotationsReportReader and YoloAnnotationsReportWriter now support annotations in polygon format as well (index plus normalized x/y polygon coordinates).

  • The Draw transformer now accepts plain BufferedImage objects as well.

  • The Image plugin for the Draw transformer now allows specifying an alpha value for transparency as well.

  • The OpexObjectLocationsReader/Writer now support per object meta-data as well.

  • Added support for tools in the ImageObjectAnnotator transformer interface. The Annotator tool lets you switch how you annotate objects (eg from bbox to polygon) while annotating. With Edit existing annotations can be modified (resizing bboxes, moving/adding vertices) and with Move annotations can be moved (dragged with mouse or moved via arrow keys).

• adams-json: The JsonObjectToMap conversion can turn JSON arrays into lists now as well (via option).

• adams-matlab: The Mat5StructToMap now handles structs with more elements than 1x1 correctly and allows the user to specify the index of a specific element to retrieve.

• adams-ml-app: added adams-docker as dependency

• adams-moa: excluding slf4j-log4j12 from moa dependency now, to avoid duplicate logger class.

• adams-nlp: excluding the outdated commons-lang3 dependency from com.kennycason:kumo-core and using version 3.7 instead explicitly.

• adams-redis: added support for authentication and experimental SSL parameters (useSSL, StartTLS, SSLVerifyMode) to the RedisConnection standalone.

• adams-rsync: upgraded rsync4j to 3.2.7-1.

• adams-spectral-3way-core: excluding log4j-slf4j-impl transitive dependency from multiway-algorithms now to avoid duplicate logger class.

• adams-spreadsheet: The Spreadsheet viewer now has a Window menu, offering to open a new viewer window or adjusting the current one to half width/height.

• adams-weka and adams-weka-lts:

  • The operations Merge, Randomize and RandomSubset that are available in the table listing the datasets now keep their last settings.

  • The Investigator now has a Window menu, offering to open a new Investigator window or adjusting the current one to half (of the screen) width/height.

• adams-yaml: upgraded snakeyaml to 1.33

• adams-xml: upgraded xstream to 1.4.20 to address CVE-2022-40151 and CVE-2022-41966.

Additions

• adams-audio: The SpeechPlaybackHandler allows playback of WAV/MP3 files and displays the content of the associated .txt file.

• adams-core:

  • The ObjectViewer sink allows the user to inspect any object with ADAMS' built-in inspection capabilities.

  • The Flow editor now has the Parse error item in the Run menu which allows to the user to paste an error output (which contains actor names) from the console. This error then gets parsed and the user can navigate to the actors (if possible).

  • The LastLineMatches file use scheme simply checks whether the last line in a text file matches the specified regular expression (useful under Linux).

  • SelectDirectory/SelectFile/EnterValue actor can be "swapped" in the Flow editor with a EnterManyValues instance now.

• adams-docker: basic support for executing docker commands. Linux/Mac only or WSL2, as it relies on the docker command-line client for execution.

• adams-imaging:

  • The ObjectAnnotationsHandler now has an OverlapRemoval cleaner, e.g., for only keeping predictions with the highest score.

  • The ImageClassificationHandler overlays label (and optional score) onto an image that was read with the specified report reader.

  • The ImageClassificationSpreadSheetReportReader loads spreadsheets with image classification predictions and stores the highest label and its score in the report.

  • The SimpleOverlay image segmentation container operation turns base image and layers into a combined image that can be used in reports or saved to disk; CompareAnnotationWithPrediction generates images from annotation vs prediction, highlighting the overlappoing, missed and additional pixels.

  • The ImageSegmentationViewer allows the viewing of ImageSegmentationContainer tokens, making it easy to toggle layers for better inspection.

  • The AnyImageToByteArray and BufferedImageToByteArray conversions turn an image into a byte array of a certain format (e.g., JPG or PNG).

  • The ByteArrayToBufferedImage generates an image container from the incoming byte array representing an image (e.g., JPG or PNG).

  • The PolygonPointAnnotator allows the user to annotate polygons by left-clicking on each vertex (ENTER accepts, ESCAPE discards).

  • The PolygonFill tool allows you to define a polygon in the ImageSegmentationAnnotator transformer that then gets filled with either foreground or background color. The polygon gets defined by left-clicking on the individual vertices.

  • Added meta-plotters for annotation outlines, labels and shapes that apply the sub-plotters subsequently.

  • Added vertex outline plotters for object annotations (bbox and polygon).

• adams-imaging-ext:

  • A new module with imaging tools that rely on external tools, like deep learning frameworks that run in docker containers.

  • DEXTR (Deep Extreme Cut) can aid the human annotator in annotating shapes for image segmentation (via transformer ImageSegmentationAnnotator).

  • OPEX can obtain object detection predictions from an OPEX-emitting docker container (via transformer ImageObjectAnnotator).

• adams-json:

  • With ImageClassificationJsonReportReader it is possible to read JSON files that contain label/score pairs generated as predictions by image classification frameworks.

  • The JsonArrayToList conversion turns a JSON array into a Java list.

  • The ListToJsonArray conversion turns a Java list into a JSON array.

• adams-matlab:

  • The DoubleMatrixToMat5Array conversion turns a double matrix into a 2-dimensional Matlab array.

  • The MapToMat5Struct conversion turns a Java map (nested maps, spreadsheets, Matlab arrays) into a Matlab struct object.

  • The Mat5ArrayToString conversion generates a human-readable string from a Matlab matrix.

  • The StringToMat5Array conversion parses a string like [1, 2; 3, 4] into a Matlab matrix.

  • The MultiMatrixOperation transformer applies a matrix operation to the matrices passing through and outputs the results.

  • The Mat5CharToString conversion turns a Char object into a string.

• adams-spreadsheet: The AutoWidthTabularSpreadSheetReader/Writer pair allows parsing/outputting of tabular text files with dynamic column widths. For reading, the column width is being determined based on the header row. For writing, the column width is adjusted to fit all content in the output.

• adams-weka and adams-weka-lts:

  • added the Remove test set action to the Investigator, which uses the RemoteTestInstances filter under the hood for removing instances of a test set from a source dataset (using a unique ID).

  • added the Soerensen-Dice metric for nominal classes (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient).

This time round we have a bigger update coming: further modules retired, added experimental module for initial OpenCV image processing support, the Matlab module got extended massively (now can create structs/matrices and populate them, not just read them), a number of dependencies were upgraded and, works now properly under Java 17 and, last but not least, simplified switching of look'n'feel from the main menu (and added support for modern FlatLaf look'n'feel).

Fixes

• adams-core:

  • The documentation of the DeleteFile standalone now states that it only deletes files.

  • The storage tab in the control panel in the Flow editor when reading a breakpoint now keeps the last select item in the table. Therefore it updates that item whenever the breakpoint is reached again.

• adams-matlab:

  • The transformer Mat5FileInfo now sorts the output when enabled.

  • The transformer Mat5StructInfo now queries the input token for the correct input type and sorting works now as expected.

• adams-spreadsheet: The SpreadSheetDisplay sink can be accessed via CallableActorScreenshot when inside a DisplayPanelManager now as well.

Changes

• adams-core:

  • Removed the following actors (which had been marked as deprecated for a long time):

    • ConditionalStandalone

    • ConditionalSource

    • ConditionalTransformer

    • ConditionalSink

  • The package adams.flow.condition.test, which was only used by these actors, has been removed as well.

  • The SelectArraySubset transformer now has an initialSelection range parameter with which you can select which items get selected to start with and an initialSearch parameter (search must be allowed) to immediately show only a subset.

  • Now uses the simple-directtory-chooser library (https://github.com/waikato-datamining/simple-directory-chooser) for providing a way of selecting directories in the user interface.

  • The GenericObjectEditor used by the Flow editor and in other locations are now remembering size and location.

  • Simplified switching of Look'n'Feel (main menu: Program > Look'n'feel) and added support for FlatLaf (https://www.formdev.com/flatlaf/).

  • The HasSize boolean condition now allows the specification of an upper bound as well.

  • The control panel, when reaching a breakpoint, now allows the user to specify a rendering limit (token and storage; underlying renderer must support that), to adjust how much data is being displayed.

  • The SelectDirectory source actor now allows the selection of multiple directories.

  • The MapToString conversion now outputs array values in a readable format.

• adams-db: upgraded hsqldb to 2.7.1 to address CVE-2022-41853

• adams-heatmap: removed deprecated filter HeatmapCropToCentroid (replaced by HeatmapCrop/CropToCentroid)

• adams-json: upgraded gson to 2.8.9 to address CVE-2022-25647

• adams-matlab:

  • upgraded mfl-core to 0.5.11

  • renamed MatlabStructInfo to Mat5StructInfo for consistency

• adams-meka: upgraded meka to 1.9.7 (to work with Java 17)

• adams-moa: upgraded MOA to 2022.10.17 (ADAMS-internal release of fork; minus deep learning classifiers/libraries)

• adams-pdf:

  • The page ranges of the PDFExtract and PDFRenderPages transformers are now unordered (i.e., indices will not get sorted).

  • Replaced the outdated Intarsys libraries with modern IcePDF (https://github.com/pcorless/icepdf).

• adams-rest: upgraded jackson-databind to 2.14.0-rc1 to address CVE-2022-42003

• adams-spectral-3way-core: upgraded multiway-algorithms to 0.0.13 and lombok to 1.18.24 to work with Java 17

• adams-spreadsheet:

  • The SpreadSheetRowBuffer transformer now allows the user to specify a monitor variable for resetting the buffer (row -> sheet).

  • The HasRows boolean condition now allows the specification of an upper bound as well.

• adams-weka and adams-weka-lts:

  • The WekaInstanceBuffer transformer now allows the user to specify a monitor variable for resetting the buffer (instance -> instances).

  • WekaPredictionsToSpreadSheet and WekaPredictionsToInstances now have useOriginalIndices on by default to avoid accidental misalignment when adding attributes from the test data.

Additions

• adams-core:

  • The EnterManyValues source can now query for arrays using the ArrayValueDefinition plugin.

  • The DeleteDir standalone can delete sub-directories matching a regular expression or itself.

  • The HasLength boolean condition can ensure a minimum/maximum length of an array.

  • The CollectionSize transformer outputs the size (ie number of stored elements) of any collection.

  • The IsArm64 boolean condition checks whether the system architecture is Arm 64-bit.

  • The DistinctValueCellRenderingCustomizer renderer uses different a background color for each distinct value in the spreadsheet.

  • The ColorProviderBasedColorGradientGenerator can be used for generating colors using a color provider (eg useful when generating images from matrices with unique values).

• adams-imaging: added the DoubleMatrixToBufferedImage conversion, which turns a heatmap into an image.

• adams-matlab:

  • added the Mat5ArrayInfo transformer to obtain information on the incoming array (eg dimensions).

  • added the Mat5ArraySubset transformer for obtaining arbitrary subsets (n-dimensional arrays) of arrays.

  • added the Mat5ArrayToDoubleMatrix conversion.

  • added the IsMat5Array boolean flow condition.

  • added the IsMat5Matrix boolean flow condition.

  • added the NewMat5Matrix source actor for creating an empty matrix.

  • added the Mat5GetMatrixElement transformer for retrieving a single element from a n-dimensional matrix.

  • added the Mat5SetMatrixElement transformer for setting a single element in a n-dimensional matrix.

  • added the NewMat5Struct source actor for creating an empty struct.

  • added the Mat5GetStructField transformer for retrieving an array by name from a struct.

  • added the Mat5SetStructField transformer for setting an array by name in a struct.

• adams-opencv: new module for image processing using JavaCV's wrapper around OpenCV.

Retired modules

The following modules have been retired and moved to https://github.com/waikato-datamining/adams-discontinued:

• adams-dex-app: due to retiring RabbitMQ modules

• adams-jython: Jython hasn't progressed beyond Python 2.7 for many years

• adams-rabbitmq: focus shifted, no longer required

• adams-rabbitmq-rats: due to retiring adams-rabbitmq

Apart from retiring a bunch of unused modules, upgrading a range of dependencies, I've switched the codebase to allow Java 11 syntax. Even though only switching to 11 for the code, the compiled binaries should work with OpenJDK 17 (the next long-term-support version of OpenJDK).

Fixes

• The MemoryMonitor standalone now stops the monitoring when the flow stops.

• The ListVariableUsage actor processor (used by Find usages in the Flow editor tree popup) now lists boolean options as well.

• EnterManyValues now restores time/date/datetime values correctly, the ListSelectionValueDefinition now can have multi-selection enabled.

• adams-imaging: The PascalVOCObjectLocationsReader report reader no longer fails with a NullPointerException if the segmented node is not present.

• adams-weka and adams-weka-lts: The WekaSpreadSheetToPredictions transformer now handles labels correctly that are only present in the predicted column.

Changes

• adams-access: upgraded jackcess to 4.0.2

• adams-compress:

  • upgraded compress-lzf to 1.1

  • upgraded zip4j to 2.11.2

  • upgraded junrar to 7.5.3

  • upgraded xz to 1.9

  • upgraded zstd-jni to 1.5.2-4

• adams-core:

  • Upgraded mysql-connector-java to 5.1.49 to address Cannot connect: java.sql.SQLException: Unknown system variable 'query_cache_size' error

  • Upgraded jclasslocator to 0.0.22 for speeding up classpath analysis

  • The CounterInit standalone and transformer now support initializing strings with a specific value

  • The Preview browser now allows fixing the preview handler, i.e., it does not get reloaded from its .props file with the next file. This is useful when using two or more preview windows with different handlers for the same file type/extension.

  • The HashSet source now allows the user to specify a conversion for converting the hashset elements into custom object types in the spreadsheet (default: string).

  • The Flow editor now records what actor handlers are being used for enclosing other actors to offer a common shortlist in the new, compact Enclose item of the tree popup menu.

  • The Make conditional action in the Flow editor now uses ConditionalTrigger/SubProcess/Tee for sources/transformers/sinks that don't have a conditional equivalent.

  • The Expression plot post-processor can handle plot containers without X values now.

• adams-db-mysql8: Upgraded mysql-connector-java to 8.0.30

• adams-excel:

  • Apache POI dependencies upgraded to 4.1.2

  • The ExcelSpreadSheetReader can use names in the sheet range now, also improved the handling of cell types (e.g., formulas and errors)

  • The ExcelSpreadSheetWriter now allows customizing the formats of time, date and date/time cells. Also handles boolean cells now.

• adams-groovy: upgraded groovy-all dependency to 3.0.8

• adams-html: upgraded jetty-server to 9.4.48.v20220622

• adams-imaging:

  • The IntersectOverUnionRatio object overlap scheme can now either use bounding boxes or polygons for the IoU calculation.

  • The JAI Resize BufferedImage transformer now has a forcePercentage option to allow upscaling images.

• adams-jython: upgraded jython-standalone to 2.7.3

• adams-matlab: The Mat5SpreadSheetReader can use names in the sheet range now

• adams-math: * The LOWESS, SavitzkyGolay and SimpleSavitzkyGolay plot post-processors can handle plot containers without X values now.

• adams-ml: The ConfusionMatrix transformer can now enumerate the labels (in case of very long labels) and outputs a second spreadsheet with the index/label relation.

• adams-net: switched to com.github.mwiede fork of jsch library

• adams-odf: The ODFSpreadSheetReader can use names in the sheet range now

• adams-pdf:

  • The SpreadSheet PDF proclet allows customizing the table width, horizontal alignment and relative column widths.

  • upgraded pdfbox to 2.0.27

• adams-python: moved Pyro4 related code into adams-pyro4 module

• adams-random: The transformers ArrayFolds and ArraySubSample now can turn off creating a deep copy of the elements before transferring them to the target array.

• adams-rats-django: updated to work with Django 4.1.x and Python 3.8+

• adams-rest: upgraded jackson-databind to 2.13.4 to address CVE-2022-42004

• adams-rsync: upgraded rsync4j-windows64 to 3.2.3-10

• adams-spreadsheet:

  • The SpreadSheetDisplay sink now allows to hide the column with the row index and use a simple header.

  • The GnumericSpreadSheetReader can use names in the sheet range now

  • The NewSpreadSheet source now can have the headers supplied via an array, which takes precedence over the comma-separated list.

• adams-webservice: upgraded Apache CXF to 3.5.3

• adams-yaml: Upgraded snakeyaml to 1.32 to address CVE-2022-25857 and CVE-2022-38752

Additions

• adams-core:

  • The ArrayDisplay sink allows displaying array elements in tabs, horizontally, vertically or in a grid.

  • Added the MapVariableIterator for iterating over key/value pairs of a map, storing key/value in user-specified variables.

  • Added the Copy (commandline) option to the Flow editor popup menu (Copy...) to allow pasting in the GenericObjectEditor.

  • Added the StringToCharArray and CharArrayToString conversions.

• adams-imaging:

  • The Regions plugin for the Subimages buffered image transformer allows the user to specify arbitrary rectangles (and their annotations) to be extracted.

  • The ImageObjectOverlapMatrix transformer generates a matrix between overlapping annotations and predictions, which can be turned into a confusion matrix.

  • Additional (non-JAI) Resize BufferedImage transformer added for resizing images.

  • The MergeOverlaps image object filter can combine multiple, overlapping objects into a single one.

  • The MergeOverlaps overlapping object removal scheme combines overlapping objects rather than removing them.

  • The DetermineOverlappingAreas transformer computes areas of overlapping objects between annotations and predictions.

• adams-pdf:

  • Added the FixedText PDF proclet that allows arbitrary text to be added (and variables get expanded automatically) without having to load a file from disk.

  • The ConfusionMatrix PDF proclet inserts a confusion matrix with better highlighting of the diagonal and (optionally) the other cell values.

• adams-pyro5: same as adams-pyro4, but uses the pyrolite library for communicating with Pyro5

• adams-spreadsheet:

  • Added the conversions SpreadSheetToRowArray and RowArrayToSpreadSheet for managing a spreadsheet as row array and apply existing array operators.

Retired modules

The following modules have been retired and moved to https://github.com/waikato-datamining/adams-discontinued:

• adams-cqengine: module retired due to no downstream projects making use of it

• adams-groovy-webservice: module retired due to com.github.groovy-wslite:groovy-wslite project being abandoned

• adams-osm: display of maps in Swing not well supported

• adams-tensorflow: module retired due to limited usefulness

• adams-deeplearning-spectral-app: module retired (no added benefit since adams-tensorflow got retired)

• adams-twitter: no real benefit of module

• adams-twitter-rats: due to adams-twitter getting retired

It's been quiet for too long in terms of updates... A lot of little things have happened, with dependencies getting upgraded to address security issues, a lot of things happening in the image annotation space, support for JDBC BLOB/CLOB reading/writing and handlers for the Preview browser for audio files.

Fixes

• The Pull up operation in the Flow editor now ensures that the actor names in the parent node are unique.

• adams-weka and adams-weka-lts: The XGBoost classifier now handles classification properly, improved speed at prediction time.

• adams-imaging:

  • IndividualImageSegmentationLayerReader now reads layers as ARGB rather than RGB.

  • IndividualImageSegmentationLayerWriter now writes proper binary PNG files for the individual layers.

  • The BlueChannelImageSegmentationWriter no longer overwrites the JPG base image.

  • The Eraser image segmentation tool now works properly when zoomed in or out.

  • The ImageSegmentationContainer can be copied now, due to implementing its own getClone() method to handle BufferedImage objects properly (can't be serialized).

Changes

• The GenericObjectEditor now displays options and help in two separate tabs rather than a split pane, which previously could make it look too squashed. Resizing its window when switching classes has also been minimized.

• The ReportField transformer is now derived from AbstractArrayProvider and can output the fields as array.

• adams-compress: upgraded zip4j to 2.9.1

• adams-imaging:

  • upgraded metadata-extractor to 2.17.0 (CVE-2022-24613, CVE-2022-24613, CVE-2022-24614)

  • The ImageObjectAnnotator now offers annotation checks to ensure that only valid annotations can be accepted.

  • The preview browser handlers BlueChannelImageHandler, GrayOrIndexedImageHandler and TrimapImageHandler allow the user now to select an image reader.

  • The ImageSegmentationContainerFilter transformer now implements InPlaceProcessing and creates a copy of the container by default.

• adams-net:

  • The SMTPConnection standalone now allows the specification of the protocols to use (eg TLSv1.2).

  • The Email preferences now allow user to specify SMTP protocols to use.

  • Upgraded javax.mail to 1.5.6

  • Upgraded commons-mail to 1.5

  • Upgraded commons-net to 3.8.0

• adams-meka: Upgraded Meka to 1.9.6

• adams-pdf: Upgraded itextpdf to 5.5.13.3

• adams-rest: Upgraded jackson-databind to 2.13.2.1 to address CVE-2020-36518

• adams-rsync: Upgraded rsync4j to 3.2.3-9

• adams-weka and adams-weka-lts: upgraded xgboost4j to 1.5.2

Additions

• The tree popup menu in the Flow editor now has a sub menu for adding the most common actors based on the user's usage.

• The FromFileHandler Preview browser handler simply points to a text file containing a setup (command-line or nested format) of the actual preview handler to use. Useful for maintaining and sharing a range of preview setups.

• The ByteArrayToJavaObject and JavaObjectToByteArray allow deserializing/serializing arbitrary objects from/to byte arrays.

• The LargeObjectDbReader source and the LargeObjectDbWriter sink allow reading/writing of BLOB/CLOB entities.

• adams-imaging:

  • Added the LocatedObjectsToReport conversion, which turns arrays of image objects back into a Report object, to be attached to an image.

  • Added a plugin architecture for object annotation overlays, which makes it easier to create more customizable overlays: ObjectAnnotations (Image viewer), ObjectAnnotationsHandler (Preview browser), ObjectAnnotationsFromReport (Draw transformer), ObjectAnnotations (object overlay).

  • The ImageObjectToVariables transformer turns the properties of the LocatedObject passing through into variables (eg, x/y/width/height).

  • The SetImageObjectMetaData transformer allows setting/updating meta-data of individual image objects (of type LocatedObject).

  • The SimpleImageSegmentationAnnotationsHandler plugin for the Preview browser allows the overlay of annotations from PNG files over the actual JPG image.

  • The ImageSegmentationAnnotationsHandler generates a more sophisticated interface with the various image segmentation layers. Though it is slower than the simple handler, it supports also the individual segmentation layers format.

  • Added reader/writer pair for grayscale image segmentation PNG files.

  • Added Switch label tool to image segmentation annotation interface, which changes the label from one to another (eg when accidentally mislabeling).

  • Added the Draw and MergeLayers image segmentation container filters.

• adams-audio:

  • The AudioFileInfoHandler in the Preview browser displays information on .mp3 and .wav files.

  • The AudioFilePlaybackHandler allows playback of .mp3 and .wav files in the Preview browser.

A lot of effort was going on behind the scenes to update or remove dependencies that were affected by to the log4j vulnerability.

Fixes

• The Preview browser now has an upper limit (defined in its props file) for how many previews will get cached, to avoid unnecessary memory being gobbeled up.

• Leaving files in the GenericObjectEditor now falls back on "." (or the current working directory) rather than resulting in a parse error.

• adams-imaging: The following plugins of the RemoveOverlappingImageObjects transformer no longer add duplicate objects to the result under certain circumstances: KeepHighestMetaDataValue, RemoveLargerRectangle, RemoveSmallerRectangle.

• adams-ml: The SimpleArffSpreadSheetReader/Writer now correctly quote/unquote attribute names.

Changes

• adams-core and adams-maven-plugin: updated logback-core/logback-classic to 1.2.10 (to address log4j)

• adams-compress: upgraded junrar to 7.4.1 to address https://github.com/advisories/GHSA-m6cj-93v6-cvr5

• adams-excel: removed unused openxml4j dependency to avoid old version of log4j being pulled in

• adams-imaging: the ImageSegmentationAnnotator now allows the application of filters to the layers (as a whole)

• adams-rest: updated jackson-databind from 2.9.8 to 2.9.10.8

• adams-spectral-3way-core: upgraded multiway-algorithms to 0.0.9

• adams-spreadsheet: added COLUMN_VALUES which returns all cell values in verbatim order.

• adams-video: removed unused org.openimaj:video-processing dependency as it pulls in old version of log4j

• adams-weka: updated Weka to 3.9.6-fork-0.0.1

• adams-xml: updated xstream to 1.4.19 to address https://x-stream.github.io/CVE-2021-43859.html

Additions

• adams-imaging:

  • The Size object finder allows filtering by size (width, height or both) using lower and upper bounds.

  • Added I/O support for the YOLO object annotation format (YoloAnnotationsReportReader, YoloAnnotationsReportWriter) and in the preview browser (YoloAnnotationsHandler).

  • Added the AddDefaultLabel object filter, which sets a default label for annotations if currently none set (to be used with ImageObjectFilter transformer).

The videos from the Weka User Conference from last year in November are now finally available.

The lead up to the Weka Conference on November 26th (https://events.waikato.ac.nz/events/2021-international-weka-user-conference/) resulted in a lot of minor bug fixes and improvements. I've also started retiring some of the modules that are no longer being used/useful.

Fixes

• Fixed issue of the Flow editor locking up (occasionally) when debugging flows; the cause was a race condition of two debugging control panels getting generated, with the incorrect one replacing the other one.

• Fixed duplicate class cache initialization, resulting in faster start-up time.

• Position and size of applications launched from main menu now get properly recorded when closing the applications.

• Reduced startup time of applications that use file choosers, avoiding unnecessary clipboard lookups.

• adams-imaging:

  • The object annotator now uses the correct zoom when initially displaying the image.

  • Image segmentation annotation no longer has the darkening bug when adding layers.

• adams-weka and adams-weka-lts:

  • Convert to Date/Nominal/String in the Weka Investigator's Preprocess panel now works with numeric attribute names as well.

  • The SimpleArffSaver and SpreadSheetSaver converters no longer use resetOptions() to initialize their options with default values.

  • The WekaCrossValidationEvaluator now stops execution properly and cleans up without causing NullPointerException.

Changes

• moved discontinued modules into adams-discontinued repo (https://github.com/waikato-datamining/adams-discontinued):

  • adams-cntk

  • adams-cntk-weka

  • adams-image-webservice

  • adams-imaging-imagemagick

  • adams-imaging-opemimaj

  • adams-jooq

  • adams-meka-webservice

  • adams-mongodb

  • adams-weka-nd4j

  • adams-weka-nd4j-lts

  • adams-weka-webservice

• The GenericObjectEditor now displays the help within the same dialog/frame to avoid help window disappearing behind other windows when opened from a modal dialog/frame. The drop-down button/menu next to the classname now has a Use default menu which reverts the current settings back to the default ones.

• adams-webservice: upgraded CXF to 3.4.5, xercesimpl to 2.12.1, jaxb to 3.0.2

• adams-bootstrapp: upgraded bootstrapp to 0.1.11

• adams-net: upgraded tika-core to 1.27, requests4j to 0.2.2

• adams-moa: upgraded MOA to 2021.07.0

• adams-core:

  • upgraded jclasslocator to 0.0.19

  • jfilechooser-bookmarks updated to 0.1.8 for better clipboard handling

  • removed DocBookProducer for generating DocBook XML from setups (not feasible for large flows)

• adams-weka and adams-weka-lts: The WekaPredictionsToInstances and WekaPredictionsToSpreadSheet transformers now handle attribute names in the test attributes option.

• adams-redis and adams-rats-redis: switched to the lettuce.io Redis Java client

Additions

• adams-weka and adams-weka-lts:

  • The LeaveOneOutByValueGenerator split generator generates pairs using the unique values from a specified attribute, with each unique value being in the test set and the remainder in the training set.

  • The MultiRowProcessor instance filter identifies rows with a row selection scheme and then processes them with a row processing scheme, e.g., for averaging multiple scans of the sample.

This time, it took a bit longer compiling this list of updates. First the 2-year old SSD in my laptop died and then parts of NZ went in and out of COVID-19 lockdown several times. And not to forget, I've been busy working remotely on a bunch of projects, with quite a bit more Python/Docker these days. As an aside, I laid the foundations for a Python-based and ADAMS-inspired workflow engine called shallowflow (https://github.com/waikato-datamining/shallowflow). But that is still at a very early stage and without a user interface for the time being.

Fixes

• adams-spreadsheet:

  • The FixedTabularSpreadSheetReader no longer skips the first line.

  • Formulas now work with Long values as well, not just Double values.

Changes

• ArraySubset is now superseded by the more flexible ArraySubsetGeneration transformer.

• The Preview browser now has support for favorite content handlers, which allows one to quickly switch between different views for the same file.

• adams-meka: Upgraded Meka to 3.9.5

• adams-r: Upgraded RSyntaxArea to 3.1.2

• adams-pdf: Upgraded pdfbox to 2.0.24

• adams-jython: Upgraded Jython to 2.7.2

• adams-rsync: Upgraded rsync4j to 3.2.3-7

• adams-weka and adams-weka-lts: Upgraded xgboost4j to xgboost4j_2.12/1.4.1

• adams-spectral-2dim-core: KennardStone now has an invert flag

• adams-weka and adams-weka-lts:

  • KennardStone unsupervised instance filter now has an invert flag

  • The SavitzkyGolay and SavitzkyGolay filters now have a -keep-attribute-names flag.

  • When selecting multiple attributes in the Weka Investigator's Preprocess panel, the Attribute summary and visualization sections now show the data for each selected attribute, not just one.

  • Entries in the history panels of the Weka Investigator can be renamed now.

• The obsolete modules adams-cntk, adams-cntk-weka and adams-cntk-weka-lts have been removed.

• adams-spreadsheet: added more string functions to the LookUpUpdate parser: substr, left, mid, right, rept, concatenate, lower[case], upper[case], trim, matches, len[gth], find, contains, replace, replaceall, substitute, str

Additions

• Added a generic framework for generating indexed splits runs from datasets, with concrete implementations for data structures (spreadsheets, Weka Instances). This framework allows to generate splits from datasets, like cross-validation or grouped cross-validation, but not the actual data, only the row indices. That way model performances from different frameworks can be compared directly.

• Added MultiMapOperation transformer, which allows operations such as: CommonKeys and Merge.

• Added the ArraySubsetGeneration transformer which has its own plugin hierarchy.

• Added the adams-db-mysql8 module for accessing MySQL 8 servers

• adams-compress: Addded the boolean conditions for byte arrays to check for compression formats: IsBzip2Compressed, IsGzipCompressed, IsRarCompressed, IsXzCompressed, IsZipCompressed, IsZstCompressed.

• adams-spreadsheet:

  • Added column finder ByExactName.

• adams-weka and adams-weka-lts:

  • Added Detrend unsupervised attribute filter (Mean, RangeBased).

  • Add the AverageSilhouetteCoefficient and MultiClustererPostProcessor clusterer post-processors.

  • Added attribute finder ByExactName.

  • Added unsupervised attribute filter MultiplicativeScatterCorrection.

  • The Weka Investigator can now generate and apply indexed splits.

  • The WekaCrossValidationEvaluator transformer now cleans up internal data structures after finishing to conserve memory.

• adams-ml: Add the MeanAbsoluteError summary statistic plugin.

• adams-spectral-2dim-core:

  • added Detrend spectrum filter (Mean, RangeBased).

  • added the SpectrumPaintletNumericField paintlet that uses a numeric field in the report that stores the color index of a color gradient generator.

  • added the SpectrumPaintletStringField paintlet that uses string extracted via regexp from a string field to determine color from color provider.

  • added the UnscramblerSpectrumReader for Unscrambler files.

  • The Evaluator transformer now has an option to discard the evaluator instance after training (e.g., when only wanting to output a trained instance).

• adams-r: added better support for R using Renjin (https://renjin.org/).

• adams-redis: new module with some basic support for the Redis in-memory database.

• adams-rats-redis: new module with some basic support for the Redis in-memory database.

• adams-imaging:

  • Added OpexObjectLocationsReader and OpexObjectLocationsWriter for reading the opex JSON object annotations (https://github.com/WaikatoLink2020/objdet-predictions-exchange-format).

  • Added PascalVOCObjectLocationsReader for reading Pascal VOC XML annotations.

• adams-tensorflow: Added the plugin TfliteModelMakerCSV for generating file-based datasets.

A bunch of changes and updates happened since last time in relation to spreadsheet handling (general and spectral data related). One thing that I've been trying to sort out was creating screenshots of tables. You can now use Send to -> Export table as image on pretty much any native ADAMS table to create a screenshot of a table.

Fixes

• The ListVariableUsage actor processor now also inspects VariableNameValuePair (and arrays) and BaseObject arrays for variables.

Changes

• The search panel received an overhaul, including an added clear button. The Preview browser now performs incremental search for a more efficient search.

• The spreadsheet handlers in the Preview browser now allow the user to select a custom cell rendering, e.g., for highlighting certain values.

• Most tables should now offer Send to capability and the new Export table as image action to export the current table as an image in various formats. With the Copy table as image, the table gets copied as image to the clipboard.

• The SpreadSheetDisplay sink can now be used in conjunction with the CallableActorScreenshot actor.

• The Preview browser supports export of tables as images now as well.

• commons-io/commons-io upgraded to 2.7

• adams-imaging:

  • The ObjectLocationsFromReport operation for the Draw transformer now allows varying the shape color as well, just like in the Preview browser.

  • Object location overlays can now fall back on the bounding box if the polygon is too small (eg from a bad mask). Also applies to the Preview browser and Draw transformer plugin.

• adams-matlab: improved handling of character cells in the MatlabArrayToSpreadSheet conversion.

• adams-net: The HttpRequest transformer now has an option for defining the mime-type of the data, default is application/octect-stream.

• adams-weka and adams-weka-lts: The WekaInstancesDisplay sink can now be used in conjunction with the CallableActorScreenshot actor now.

• adams-spreadsheet: The SpreadSheetInsertRow transformer can interpret the value now as blank or comma-separated values as well. The value(s) can now be forced to be STRING as well.

• adams-spectral-2dim-core: the JSON format for spectra has been overhauls, using separate arrays for wave numbers and amplitudes now to reduce file size. Also, either the full report or specific reference+meta-data values can be written/read.

Additions

• The HeatmapCellRenderingCustomizer cell rendering customizer for spreadsheet tables allows one to color in the background of numeric cells based on the global min/max and the specified color generator. Makes it easier to spot extreme values.

• The HasElements boolean flow condition checks whether the incoming object is an array and has at least specified number of elements.

• The HasSize boolean flow condition checks whether the incoming collection has at least the specified size.

• The ReportArrayToMap conversion turns the a report array into a map using the specified report field as key.

• The MergeReportFromMap transformer allows the reports passing through (either Report or MutableReportHandler objects) to be merged with reports available from a map in storage. This way, meta-data can be read separately and stored in a map in storage and then easily attached.

• adams-imaging:

  • Added general support for point annotations, which can be used for pose estimation.

  • Added support for reading/writing DeepLabCut CSV annotation files

• adams-spectral-2dim-core:

  • The Flatliner outlier detector picks up on spectra that only consist of one value.

  • The SpreadSheetColumnsToSpectra conversion generates multiple spectra from a spreadsheet, more flexible than SpreadSheetColumnToSpectrum. The SpreadSheetRowsToSpectra conversion works similar, but row-wise instead of column-wise.

  • The SpreadSheetColumnsToSampleData conversion generates sample data objects (= reports) from columns from a spreadsheet. The SpreadSheetRowsToSampleData conversion does the same using rows from a spreadsheet.

  • The SampleDataArrayToMap conversion generates a map from the array, using the sample ID as key for the map.

  • The SpreadSheetRowGenerator transformer allows you to generate spreadsheet rows from spectra using various generators.

• adams-heatmap: Added the MultiHeatmapOperation transformer, which allows applying an operation to an array of heatmaps.

• adams-spreadsheet:

  • The SpreadSheetUseRowAsHeader conversion replaces the values in the header row with the ones from the specified data row.

  • The SpreadSheetColumnsToReport and SpreadSheetRowsToReport conversions convert columns/rows in a spreadsheet containing meta-data into Report objects.