Updates 2022/11/11

This time round we have a bigger update coming: further modules retired, added experimental module for initial OpenCV image processing support, the Matlab module got extended massively (now can create structs/matrices and populate them, not just read them), a number of dependencies were upgraded and, works now properly under Java 17 and, last but not least, simplified switching of look'n'feel from the main menu (and added support for modern FlatLaf look'n'feel).

Fixes

  • adams-core:

    • The documentation of the DeleteFile standalone now states that it only deletes files.

    • The storage tab in the control panel in the Flow editor when reading a breakpoint now keeps the last select item in the table. Therefore it updates that item whenever the breakpoint is reached again.

  • adams-matlab:

    • The transformer Mat5FileInfo now sorts the output when enabled.

    • The transformer Mat5StructInfo now queries the input token for the correct input type and sorting works now as expected.

  • adams-spreadsheet: The SpreadSheetDisplay sink can be accessed via CallableActorScreenshot when inside a DisplayPanelManager now as well.

Changes

  • adams-core:

    • Removed the following actors (which had been marked as deprecated for a long time):

      • ConditionalStandalone

      • ConditionalSource

      • ConditionalTransformer

      • ConditionalSink

    • The package adams.flow.condition.test, which was only used by these actors, has been removed as well.

    • The SelectArraySubset transformer now has an initialSelection range parameter with which you can select which items get selected to start with and an initialSearch parameter (search must be allowed) to immediately show only a subset.

    • Now uses the simple-directtory-chooser library (https://github.com/waikato-datamining/simple-directory-chooser) for providing a way of selecting directories in the user interface.

    • The GenericObjectEditor used by the Flow editor and in other locations are now remembering size and location.

    • Simplified switching of Look'n'Feel (main menu: Program > Look'n'feel) and added support for FlatLaf (https://www.formdev.com/flatlaf/).

    • The HasSize boolean condition now allows the specification of an upper bound as well.

    • The control panel, when reaching a breakpoint, now allows the user to specify a rendering limit (token and storage; underlying renderer must support that), to adjust how much data is being displayed.

    • The SelectDirectory source actor now allows the selection of multiple directories.

    • The MapToString conversion now outputs array values in a readable format.

  • adams-db: upgraded hsqldb to 2.7.1 to address CVE-2022-41853

  • adams-heatmap: removed deprecated filter HeatmapCropToCentroid (replaced by HeatmapCrop/CropToCentroid)

  • adams-json: upgraded gson to 2.8.9 to address CVE-2022-25647

  • adams-matlab:

    • upgraded mfl-core to 0.5.11

    • renamed MatlabStructInfo to Mat5StructInfo for consistency

  • adams-meka: upgraded meka to 1.9.7 (to work with Java 17)

  • adams-moa: upgraded MOA to 2022.10.17 (ADAMS-internal release of fork; minus deep learning classifiers/libraries)

  • adams-pdf:

    • The page ranges of the PDFExtract and PDFRenderPages transformers are now unordered (i.e., indices will not get sorted).

    • Replaced the outdated Intarsys libraries with modern IcePDF (https://github.com/pcorless/icepdf).

  • adams-rest: upgraded jackson-databind to 2.14.0-rc1 to address CVE-2022-42003

  • adams-spectral-3way-core: upgraded multiway-algorithms to 0.0.13 and lombok to 1.18.24 to work with Java 17

  • adams-spreadsheet:

    • The SpreadSheetRowBuffer transformer now allows the user to specify a monitor variable for resetting the buffer (row -> sheet).

    • The HasRows boolean condition now allows the specification of an upper bound as well.

  • adams-weka and adams-weka-lts:

    • The WekaInstanceBuffer transformer now allows the user to specify a monitor variable for resetting the buffer (instance -> instances).

    • WekaPredictionsToSpreadSheet and WekaPredictionsToInstances now have useOriginalIndices on by default to avoid accidental misalignment when adding attributes from the test data.

Additions

  • adams-core:

    • The EnterManyValues source can now query for arrays using the ArrayValueDefinition plugin.

    • The DeleteDir standalone can delete sub-directories matching a regular expression or itself.

    • The HasLength boolean condition can ensure a minimum/maximum length of an array.

    • The CollectionSize transformer outputs the size (ie number of stored elements) of any collection.

    • The IsArm64 boolean condition checks whether the system architecture is Arm 64-bit.

    • The DistinctValueCellRenderingCustomizer renderer uses different a background color for each distinct value in the spreadsheet.

    • The ColorProviderBasedColorGradientGenerator can be used for generating colors using a color provider (eg useful when generating images from matrices with unique values).

  • adams-imaging: added the DoubleMatrixToBufferedImage conversion, which turns a heatmap into an image.

  • adams-matlab:

    • added the Mat5ArrayInfo transformer to obtain information on the incoming array (eg dimensions).

    • added the Mat5ArraySubset transformer for obtaining arbitrary subsets (n-dimensional arrays) of arrays.

    • added the Mat5ArrayToDoubleMatrix conversion.

    • added the IsMat5Array boolean flow condition.

    • added the IsMat5Matrix boolean flow condition.

    • added the NewMat5Matrix source actor for creating an empty matrix.

    • added the Mat5GetMatrixElement transformer for retrieving a single element from a n-dimensional matrix.

    • added the Mat5SetMatrixElement transformer for setting a single element in a n-dimensional matrix.

    • added the NewMat5Struct source actor for creating an empty struct.

    • added the Mat5GetStructField transformer for retrieving an array by name from a struct.

    • added the Mat5SetStructField transformer for setting an array by name in a struct.

  • adams-opencv: new module for image processing using JavaCV's wrapper around OpenCV.

Retired modules

The following modules have been retired and moved to https://github.com/waikato-datamining/adams-discontinued:

  • adams-dex-app: due to retiring RabbitMQ modules

  • adams-jython: Jython hasn't progressed beyond Python 2.7 for many years

  • adams-rabbitmq: focus shifted, no longer required

  • adams-rabbitmq-rats: due to retiring adams-rabbitmq

Updates 2022/10/06

Apart from retiring a bunch of unused modules, upgrading a range of dependencies, I've switched the codebase to allow Java 11 syntax. Even though only switching to 11 for the code, the compiled binaries should work with OpenJDK 17 (the next long-term-support version of OpenJDK).

Fixes

  • The MemoryMonitor standalone now stops the monitoring when the flow stops.

  • The ListVariableUsage actor processor (used by Find usages in the Flow editor tree popup) now lists boolean options as well.

  • EnterManyValues now restores time/date/datetime values correctly, the ListSelectionValueDefinition now can have multi-selection enabled.

  • adams-imaging: The PascalVOCObjectLocationsReader report reader no longer fails with a NullPointerException if the segmented node is not present.

  • adams-weka and adams-weka-lts: The WekaSpreadSheetToPredictions transformer now handles labels correctly that are only present in the predicted column.

Changes

  • adams-access: upgraded jackcess to 4.0.2

  • adams-compress:

    • upgraded compress-lzf to 1.1

    • upgraded zip4j to 2.11.2

    • upgraded junrar to 7.5.3

    • upgraded xz to 1.9

    • upgraded zstd-jni to 1.5.2-4

  • adams-core:

    • Upgraded mysql-connector-java to 5.1.49 to address Cannot connect: java.sql.SQLException: Unknown system variable 'query_cache_size' error

    • Upgraded jclasslocator to 0.0.22 for speeding up classpath analysis

    • The CounterInit standalone and transformer now support initializing strings with a specific value

    • The Preview browser now allows fixing the preview handler, i.e., it does not get reloaded from its .props file with the next file. This is useful when using two or more preview windows with different handlers for the same file type/extension.

    • The HashSet source now allows the user to specify a conversion for converting the hashset elements into custom object types in the spreadsheet (default: string).

    • The Flow editor now records what actor handlers are being used for enclosing other actors to offer a common shortlist in the new, compact Enclose item of the tree popup menu.

    • The Make conditional action in the Flow editor now uses ConditionalTrigger/SubProcess/Tee for sources/transformers/sinks that don't have a conditional equivalent.

    • The Expression plot post-processor can handle plot containers without X values now.

  • adams-db-mysql8: Upgraded mysql-connector-java to 8.0.30

  • adams-excel:

    • Apache POI dependencies upgraded to 4.1.2

    • The ExcelSpreadSheetReader can use names in the sheet range now, also improved the handling of cell types (e.g., formulas and errors)

    • The ExcelSpreadSheetWriter now allows customizing the formats of time, date and date/time cells. Also handles boolean cells now.

  • adams-groovy: upgraded groovy-all dependency to 3.0.8

  • adams-html: upgraded jetty-server to 9.4.48.v20220622

  • adams-imaging:

    • The IntersectOverUnionRatio object overlap scheme can now either use bounding boxes or polygons for the IoU calculation.

    • The JAI Resize BufferedImage transformer now has a forcePercentage option to allow upscaling images.

  • adams-jython: upgraded jython-standalone to 2.7.3

  • adams-matlab: The Mat5SpreadSheetReader can use names in the sheet range now

  • adams-math: * The LOWESS, SavitzkyGolay and SimpleSavitzkyGolay plot post-processors can handle plot containers without X values now.

  • adams-ml: The ConfusionMatrix transformer can now enumerate the labels (in case of very long labels) and outputs a second spreadsheet with the index/label relation.

  • adams-net: switched to com.github.mwiede fork of jsch library

  • adams-odf: The ODFSpreadSheetReader can use names in the sheet range now

  • adams-pdf:

    • The SpreadSheet PDF proclet allows customizing the table width, horizontal alignment and relative column widths.

    • upgraded pdfbox to 2.0.27

  • adams-python: moved Pyro4 related code into adams-pyro4 module

  • adams-random: The transformers ArrayFolds and ArraySubSample now can turn off creating a deep copy of the elements before transferring them to the target array.

  • adams-rats-django: updated to work with Django 4.1.x and Python 3.8+

  • adams-rest: upgraded jackson-databind to 2.13.4 to address CVE-2022-42004

  • adams-rsync: upgraded rsync4j-windows64 to 3.2.3-10

  • adams-spreadsheet:

    • The SpreadSheetDisplay sink now allows to hide the column with the row index and use a simple header.

    • The GnumericSpreadSheetReader can use names in the sheet range now

    • The NewSpreadSheet source now can have the headers supplied via an array, which takes precedence over the comma-separated list.

  • adams-webservice: upgraded Apache CXF to 3.5.3

  • adams-yaml: Upgraded snakeyaml to 1.32 to address CVE-2022-25857 and CVE-2022-38752

Additions

  • adams-core:

    • The ArrayDisplay sink allows displaying array elements in tabs, horizontally, vertically or in a grid.

    • Added the MapVariableIterator for iterating over key/value pairs of a map, storing key/value in user-specified variables.

    • Added the Copy (commandline) option to the Flow editor popup menu (Copy...) to allow pasting in the GenericObjectEditor.

    • Added the StringToCharArray and CharArrayToString conversions.

  • adams-imaging:

    • The Regions plugin for the Subimages buffered image transformer allows the user to specify arbitrary rectangles (and their annotations) to be extracted.

    • The ImageObjectOverlapMatrix transformer generates a matrix between overlapping annotations and predictions, which can be turned into a confusion matrix.

    • Additional (non-JAI) Resize BufferedImage transformer added for resizing images.

    • The MergeOverlaps image object filter can combine multiple, overlapping objects into a single one.

    • The MergeOverlaps overlapping object removal scheme combines overlapping objects rather than removing them.

    • The DetermineOverlappingAreas transformer computes areas of overlapping objects between annotations and predictions.

  • adams-pdf:

    • Added the FixedText PDF proclet that allows arbitrary text to be added (and variables get expanded automatically) without having to load a file from disk.

    • The ConfusionMatrix PDF proclet inserts a confusion matrix with better highlighting of the diagonal and (optionally) the other cell values.

  • adams-pyro5: same as adams-pyro4, but uses the pyrolite library for communicating with Pyro5

  • adams-spreadsheet:

    • Added the conversions SpreadSheetToRowArray and RowArrayToSpreadSheet for managing a spreadsheet as row array and apply existing array operators.

Retired modules

The following modules have been retired and moved to https://github.com/waikato-datamining/adams-discontinued:

  • adams-cqengine: module retired due to no downstream projects making use of it

  • adams-groovy-webservice: module retired due to com.github.groovy-wslite:groovy-wslite project being abandoned

  • adams-osm: display of maps in Swing not well supported

  • adams-tensorflow: module retired due to limited usefulness

  • adams-deeplearning-spectral-app: module retired (no added benefit since adams-tensorflow got retired)

  • adams-twitter: no real benefit of module

  • adams-twitter-rats: due to adams-twitter getting retired

Updates 2022/06/23

It's been quiet for too long in terms of updates... A lot of little things have happened, with dependencies getting upgraded to address security issues, a lot of things happening in the image annotation space, support for JDBC BLOB/CLOB reading/writing and handlers for the Preview browser for audio files.

Fixes

  • The Pull up operation in the Flow editor now ensures that the actor names in the parent node are unique.

  • adams-weka and adams-weka-lts: The XGBoost classifier now handles classification properly, improved speed at prediction time.

  • adams-imaging:

    • IndividualImageSegmentationLayerReader now reads layers as ARGB rather than RGB.

    • IndividualImageSegmentationLayerWriter now writes proper binary PNG files for the individual layers.

    • The BlueChannelImageSegmentationWriter no longer overwrites the JPG base image.

    • The Eraser image segmentation tool now works properly when zoomed in or out.

    • The ImageSegmentationContainer can be copied now, due to implementing its own getClone() method to handle BufferedImage objects properly (can't be serialized).

Changes

  • The GenericObjectEditor now displays options and help in two separate tabs rather than a split pane, which previously could make it look too squashed. Resizing its window when switching classes has also been minimized.

  • The ReportField transformer is now derived from AbstractArrayProvider and can output the fields as array.

  • adams-compress: upgraded zip4j to 2.9.1

  • adams-imaging:

    • upgraded metadata-extractor to 2.17.0 (CVE-2022-24613, CVE-2022-24613, CVE-2022-24614)

    • The ImageObjectAnnotator now offers annotation checks to ensure that only valid annotations can be accepted.

    • The preview browser handlers BlueChannelImageHandler, GrayOrIndexedImageHandler and TrimapImageHandler allow the user now to select an image reader.

    • The ImageSegmentationContainerFilter transformer now implements InPlaceProcessing and creates a copy of the container by default.

  • adams-net:

    • The SMTPConnection standalone now allows the specification of the protocols to use (eg TLSv1.2).

    • The Email preferences now allow user to specify SMTP protocols to use.

    • Upgraded javax.mail to 1.5.6

    • Upgraded commons-mail to 1.5

    • Upgraded commons-net to 3.8.0

  • adams-meka: Upgraded Meka to 1.9.6

  • adams-pdf: Upgraded itextpdf to 5.5.13.3

  • adams-rest: Upgraded jackson-databind to 2.13.2.1 to address CVE-2020-36518

  • adams-rsync: Upgraded rsync4j to 3.2.3-9

  • adams-weka and adams-weka-lts: upgraded xgboost4j to 1.5.2

Additions

  • The tree popup menu in the Flow editor now has a sub menu for adding the most common actors based on the user's usage.

  • The FromFileHandler Preview browser handler simply points to a text file containing a setup (command-line or nested format) of the actual preview handler to use. Useful for maintaining and sharing a range of preview setups.

  • The ByteArrayToJavaObject and JavaObjectToByteArray allow deserializing/serializing arbitrary objects from/to byte arrays.

  • The LargeObjectDbReader source and the LargeObjectDbWriter sink allow reading/writing of BLOB/CLOB entities.

  • adams-imaging:

    • Added the LocatedObjectsToReport conversion, which turns arrays of image objects back into a Report object, to be attached to an image.

    • Added a plugin architecture for object annotation overlays, which makes it easier to create more customizable overlays: ObjectAnnotations (Image viewer), ObjectAnnotationsHandler (Preview browser), ObjectAnnotationsFromReport (Draw transformer), ObjectAnnotations (object overlay).

    • The ImageObjectToVariables transformer turns the properties of the LocatedObject passing through into variables (eg, x/y/width/height).

    • The SetImageObjectMetaData transformer allows setting/updating meta-data of individual image objects (of type LocatedObject).

    • The SimpleImageSegmentationAnnotationsHandler plugin for the Preview browser allows the overlay of annotations from PNG files over the actual JPG image.

    • The ImageSegmentationAnnotationsHandler generates a more sophisticated interface with the various image segmentation layers. Though it is slower than the simple handler, it supports also the individual segmentation layers format.

    • Added reader/writer pair for grayscale image segmentation PNG files.

    • Added Switch label tool to image segmentation annotation interface, which changes the label from one to another (eg when accidentally mislabeling).

    • Added the Draw and MergeLayers image segmentation container filters.

  • adams-audio:

    • The AudioFileInfoHandler in the Preview browser displays information on .mp3 and .wav files.

    • The AudioFilePlaybackHandler allows playback of .mp3 and .wav files in the Preview browser.

Updates 2022/02/04

A lot of effort was going on behind the scenes to update or remove dependencies that were affected by to the log4j vulnerability.

Fixes

  • The Preview browser now has an upper limit (defined in its props file) for how many previews will get cached, to avoid unnecessary memory being gobbeled up.

  • Leaving files in the GenericObjectEditor now falls back on "." (or the current working directory) rather than resulting in a parse error.

  • adams-imaging: The following plugins of the RemoveOverlappingImageObjects transformer no longer add duplicate objects to the result under certain circumstances: KeepHighestMetaDataValue, RemoveLargerRectangle, RemoveSmallerRectangle.

  • adams-ml: The SimpleArffSpreadSheetReader/Writer now correctly quote/unquote attribute names.

Changes

  • adams-core and adams-maven-plugin: updated logback-core/logback-classic to 1.2.10 (to address log4j)

  • adams-compress: upgraded junrar to 7.4.1 to address https://github.com/advisories/GHSA-m6cj-93v6-cvr5

  • adams-excel: removed unused openxml4j dependency to avoid old version of log4j being pulled in

  • adams-imaging: the ImageSegmentationAnnotator now allows the application of filters to the layers (as a whole)

  • adams-rest: updated jackson-databind from 2.9.8 to 2.9.10.8

  • adams-spectral-3way-core: upgraded multiway-algorithms to 0.0.9

  • adams-spreadsheet: added COLUMN_VALUES which returns all cell values in verbatim order.

  • adams-video: removed unused org.openimaj:video-processing dependency as it pulls in old version of log4j

  • adams-weka: updated Weka to 3.9.6-fork-0.0.1

  • adams-xml: updated xstream to 1.4.19 to address https://x-stream.github.io/CVE-2021-43859.html

Additions

  • adams-imaging:

    • The Size object finder allows filtering by size (width, height or both) using lower and upper bounds.

    • Added I/O support for the YOLO object annotation format (YoloAnnotationsReportReader, YoloAnnotationsReportWriter) and in the preview browser (YoloAnnotationsHandler).

    • Added the AddDefaultLabel object filter, which sets a default label for annotations if currently none set (to be used with ImageObjectFilter transformer).

Updates 2021/12/02

The lead up to the Weka Conference on November 26th (https://events.waikato.ac.nz/events/2021-international-weka-user-conference/) resulted in a lot of minor bug fixes and improvements. I've also started retiring some of the modules that are no longer being used/useful.

Fixes

  • Fixed issue of the Flow editor locking up (occasionally) when debugging flows; the cause was a race condition of two debugging control panels getting generated, with the incorrect one replacing the other one.

  • Fixed duplicate class cache initialization, resulting in faster start-up time.

  • Position and size of applications launched from main menu now get properly recorded when closing the applications.

  • Reduced startup time of applications that use file choosers, avoiding unnecessary clipboard lookups.

  • adams-imaging:

    • The object annotator now uses the correct zoom when initially displaying the image.

    • Image segmentation annotation no longer has the darkening bug when adding layers.

  • adams-weka and adams-weka-lts:

    • Convert to Date/Nominal/String in the Weka Investigator's Preprocess panel now works with numeric attribute names as well.

    • The SimpleArffSaver and SpreadSheetSaver converters no longer use resetOptions() to initialize their options with default values.

    • The WekaCrossValidationEvaluator now stops execution properly and cleans up without causing NullPointerException.

Changes

  • moved discontinued modules into adams-discontinued repo (https://github.com/waikato-datamining/adams-discontinued):

    • adams-cntk

    • adams-cntk-weka

    • adams-image-webservice

    • adams-imaging-imagemagick

    • adams-imaging-opemimaj

    • adams-jooq

    • adams-meka-webservice

    • adams-mongodb

    • adams-weka-nd4j

    • adams-weka-nd4j-lts

    • adams-weka-webservice

  • The GenericObjectEditor now displays the help within the same dialog/frame to avoid help window disappearing behind other windows when opened from a modal dialog/frame. The drop-down button/menu next to the classname now has a Use default menu which reverts the current settings back to the default ones.

  • adams-webservice: upgraded CXF to 3.4.5, xercesimpl to 2.12.1, jaxb to 3.0.2

  • adams-bootstrapp: upgraded bootstrapp to 0.1.11

  • adams-net: upgraded tika-core to 1.27, requests4j to 0.2.2

  • adams-moa: upgraded MOA to 2021.07.0

  • adams-core:

    • upgraded jclasslocator to 0.0.19

    • jfilechooser-bookmarks updated to 0.1.8 for better clipboard handling

    • removed DocBookProducer for generating DocBook XML from setups (not feasible for large flows)

  • adams-weka and adams-weka-lts: The WekaPredictionsToInstances and WekaPredictionsToSpreadSheet transformers now handle attribute names in the test attributes option.

  • adams-redis and adams-rats-redis: switched to the lettuce.io Redis Java client

Additions

  • adams-weka and adams-weka-lts:

    • The LeaveOneOutByValueGenerator split generator generates pairs using the unique values from a specified attribute, with each unique value being in the test set and the remainder in the training set.

    • The MultiRowProcessor instance filter identifies rows with a row selection scheme and then processes them with a row processing scheme, e.g., for averaging multiple scans of the sample.

Updates 2021-10-15

This time, it took a bit longer compiling this list of updates. First the 2-year old SSD in my laptop died and then parts of NZ went in and out of COVID-19 lockdown several times. And not to forget, I've been busy working remotely on a bunch of projects, with quite a bit more Python/Docker these days. As an aside, I laid the foundations for a Python-based and ADAMS-inspired workflow engine called shallowflow (https://github.com/waikato-datamining/shallowflow). But that is still at a very early stage and without a user interface for the time being.

Fixes

  • adams-spreadsheet:

    • The FixedTabularSpreadSheetReader no longer skips the first line.

    • Formulas now work with Long values as well, not just Double values.

Changes

  • ArraySubset is now superseded by the more flexible ArraySubsetGeneration transformer.

  • The Preview browser now has support for favorite content handlers, which allows one to quickly switch between different views for the same file.

  • adams-meka: Upgraded Meka to 3.9.5

  • adams-r: Upgraded RSyntaxArea to 3.1.2

  • adams-pdf: Upgraded pdfbox to 2.0.24

  • adams-jython: Upgraded Jython to 2.7.2

  • adams-rsync: Upgraded rsync4j to 3.2.3-7

  • adams-weka and adams-weka-lts: Upgraded xgboost4j to xgboost4j_2.12/1.4.1

  • adams-spectral-2dim-core: KennardStone now has an invert flag

  • adams-weka and adams-weka-lts:

    • KennardStone unsupervised instance filter now has an invert flag

    • The SavitzkyGolay and SavitzkyGolay filters now have a -keep-attribute-names flag.

    • When selecting multiple attributes in the Weka Investigator's Preprocess panel, the Attribute summary and visualization sections now show the data for each selected attribute, not just one.

    • Entries in the history panels of the Weka Investigator can be renamed now.

  • The obsolete modules adams-cntk, adams-cntk-weka and adams-cntk-weka-lts have been removed.

  • adams-spreadsheet: added more string functions to the LookUpUpdate parser: substr, left, mid, right, rept, concatenate, lower[case], upper[case], trim, matches, len[gth], find, contains, replace, replaceall, substitute, str

Additions

  • Added a generic framework for generating indexed splits runs from datasets, with concrete implementations for data structures (spreadsheets, Weka Instances). This framework allows to generate splits from datasets, like cross-validation or grouped cross-validation, but not the actual data, only the row indices. That way model performances from different frameworks can be compared directly.

  • Added MultiMapOperation transformer, which allows operations such as: CommonKeys and Merge.

  • Added the ArraySubsetGeneration transformer which has its own plugin hierarchy.

  • Added the adams-db-mysql8 module for accessing MySQL 8 servers

  • adams-compress: Addded the boolean conditions for byte arrays to check for compression formats: IsBzip2Compressed, IsGzipCompressed, IsRarCompressed, IsXzCompressed, IsZipCompressed, IsZstCompressed.

  • adams-spreadsheet:

    • Added column finder ByExactName.

  • adams-weka and adams-weka-lts:

    • Added Detrend unsupervised attribute filter (Mean, RangeBased).

    • Add the AverageSilhouetteCoefficient and MultiClustererPostProcessor clusterer post-processors.

    • Added attribute finder ByExactName.

    • Added unsupervised attribute filter MultiplicativeScatterCorrection.

    • The Weka Investigator can now generate and apply indexed splits.

    • The WekaCrossValidationEvaluator transformer now cleans up internal data structures after finishing to conserve memory.

  • adams-ml: Add the MeanAbsoluteError summary statistic plugin.

  • adams-spectral-2dim-core:

    • added Detrend spectrum filter (Mean, RangeBased).

    • added the SpectrumPaintletNumericField paintlet that uses a numeric field in the report that stores the color index of a color gradient generator.

    • added the SpectrumPaintletStringField paintlet that uses string extracted via regexp from a string field to determine color from color provider.

    • added the UnscramblerSpectrumReader for Unscrambler files.

    • The Evaluator transformer now has an option to discard the evaluator instance after training (e.g., when only wanting to output a trained instance).

  • adams-r: added better support for R using Renjin (https://renjin.org/).

  • adams-redis: new module with some basic support for the Redis in-memory database.

  • adams-rats-redis: new module with some basic support for the Redis in-memory database.

  • adams-imaging:

  • adams-tensorflow: Added the plugin TfliteModelMakerCSV for generating file-based datasets.

Updates 2021/05/28

A bunch of changes and updates happened since last time in relation to spreadsheet handling (general and spectral data related). One thing that I've been trying to sort out was creating screenshots of tables. You can now use Send to -> Export table as image on pretty much any native ADAMS table to create a screenshot of a table.

Fixes

  • The ListVariableUsage actor processor now also inspects VariableNameValuePair (and arrays) and BaseObject arrays for variables.

Changes

  • The search panel received an overhaul, including an added clear button. The Preview browser now performs incremental search for a more efficient search.

  • The spreadsheet handlers in the Preview browser now allow the user to select a custom cell rendering, e.g., for highlighting certain values.

  • Most tables should now offer Send to capability and the new Export table as image action to export the current table as an image in various formats. With the Copy table as image, the table gets copied as image to the clipboard.

  • The SpreadSheetDisplay sink can now be used in conjunction with the CallableActorScreenshot actor.

  • The Preview browser supports export of tables as images now as well.

  • commons-io/commons-io upgraded to 2.7

  • adams-imaging:

    • The ObjectLocationsFromReport operation for the Draw transformer now allows varying the shape color as well, just like in the Preview browser.

    • Object location overlays can now fall back on the bounding box if the polygon is too small (eg from a bad mask). Also applies to the Preview browser and Draw transformer plugin.

  • adams-matlab: improved handling of character cells in the MatlabArrayToSpreadSheet conversion.

  • adams-net: The HttpRequest transformer now has an option for defining the mime-type of the data, default is application/octect-stream.

  • adams-weka and adams-weka-lts: The WekaInstancesDisplay sink can now be used in conjunction with the CallableActorScreenshot actor now.

  • adams-spreadsheet: The SpreadSheetInsertRow transformer can interpret the value now as blank or comma-separated values as well. The value(s) can now be forced to be STRING as well.

  • adams-spectral-2dim-core: the JSON format for spectra has been overhauls, using separate arrays for wave numbers and amplitudes now to reduce file size. Also, either the full report or specific reference+meta-data values can be written/read.

Additions

  • The HeatmapCellRenderingCustomizer cell rendering customizer for spreadsheet tables allows one to color in the background of numeric cells based on the global min/max and the specified color generator. Makes it easier to spot extreme values.

  • The HasElements boolean flow condition checks whether the incoming object is an array and has at least specified number of elements.

  • The HasSize boolean flow condition checks whether the incoming collection has at least the specified size.

  • The ReportArrayToMap conversion turns the a report array into a map using the specified report field as key.

  • The MergeReportFromMap transformer allows the reports passing through (either Report or MutableReportHandler objects) to be merged with reports available from a map in storage. This way, meta-data can be read separately and stored in a map in storage and then easily attached.

  • adams-imaging:

    • Added general support for point annotations, which can be used for pose estimation.

    • Added support for reading/writing DeepLabCut CSV annotation files

  • adams-spectral-2dim-core:

    • The Flatliner outlier detector picks up on spectra that only consist of one value.

    • The SpreadSheetColumnsToSpectra conversion generates multiple spectra from a spreadsheet, more flexible than SpreadSheetColumnToSpectrum. The SpreadSheetRowsToSpectra conversion works similar, but row-wise instead of column-wise.

    • The SpreadSheetColumnsToSampleData conversion generates sample data objects (= reports) from columns from a spreadsheet. The SpreadSheetRowsToSampleData conversion does the same using rows from a spreadsheet.

    • The SampleDataArrayToMap conversion generates a map from the array, using the sample ID as key for the map.

    • The SpreadSheetRowGenerator transformer allows you to generate spreadsheet rows from spectra using various generators.

  • adams-heatmap: Added the MultiHeatmapOperation transformer, which allows applying an operation to an array of heatmaps.

  • adams-spreadsheet:

    • The SpreadSheetUseRowAsHeader conversion replaces the values in the header row with the ones from the specified data row.

    • The SpreadSheetColumnsToReport and SpreadSheetRowsToReport conversions convert columns/rows in a spreadsheet containing meta-data into Report objects.

Updates 2021/04/12

More work was done on the adams-matlab module and on image annotation support.

Fixes

  • The SelectArraySubset transformer now updates its JList correctly when getting subsequently called, not just the first time round.

  • Very long cell values in tables (> 1000 characters) now get excluded from calculating the optimal column width to avoid slow calculation and extremely wide tables.

  • adams-meta: The InactiveStandalone/Source/Transformer/Sink actors now copy the actors provided to the constructors rather than just using the reference.

  • adams-spreadsheet: Formulas in spreadsheets now resolve values of cell references to their native value, not just trying to convert it to double.

  • adams-weka and adams-weka-lts:

    • The WekaAttributeSelection transformer no longer puts the ranked order of attributes back into sorted order.

    • Changing the attribute weights in an Instances table now gets applied to the correct column when displaying the instance weights column.

    • Exporting the output to PDF no longer duplicates the content of text files (due to two exporters available).

    • The Data generator source in the Weka Investigator works again.

    • String and relational values no longer get rendered as HTML as this turned out to be too slow when displaying a table.

Changes

  • adams-weka: upgraded multisearch-weka-package to 2021.2.17

  • adams-spreadsheet: deprecated SpreadSheetCommonIDs, SpreadSheetDifference in favor of MultiSpreadSheetOperation

  • adams-timeseries: upgraded timeseriesForecasting Weka package to 1.1.27

  • adams-matlab: the Mat5SpreadSheetReader can now retrieve arrays from struct objects as well

  • adams-weka and adams-weka-lts:

    • Attribute weights now get displayed in the header row when showing weights.

    • The Revert action in the Weka Investigator now monitors files for changes (last modified timestamp) and gets enabled when either the file on disk got changed or it has been modified in the Investigator itself.

    • The SimpleArffLoader and SimpleArffSaver now allow the user to specify the file encoding.

Additions

  • The MapToStorageValues transformer transfers a map object into internal storage.

  • adams-imaging:

    • The ConfusionMatrix plugin for the ImageSegmentationContainerOperation transformer generates a confusion matrix of the annotated pixels versus predicted ones and the CountPixels one simply outputs a spreasdheet with counts per layer (of non-black pixels).

    • The IndexedPNGImageHandler for the Preview browser allows overriding the colors of PNG files that use an indexed color palette, e.g., when output by image segmentation models.

    • The BlueChannelImageHandler (using the new BlueChannelColorizer image transformer under the hood), interprets the values in the blue channel of an image as color indices and applies the colors from the supplied color provider accordingly.

  • adams-spreadsheet: added the MultiSpreadSheetOperation transformer which applies the specified operation to the incoming spreadsheet array (CommonIDs, Sum, Difference, Merge).

  • The HasProperty boolean condition can be used to check whether objects have a certain property name, e.g., when updating object values via the UpdateProperty transformer.

  • adams-matlab:

    • The MatlabStructToMap conversion turns a Matlab struct object into a map (field/array relation).

    • The IsMatlabStruct boolean condition checks whether the token represents a Matlab struct object.

    • The MatlabStructInfo transformer allows outputting information on a Matlab struct object.

    • With the flow adams-matlab-inspect_file.flow you can inspect .mat files interactively.

  • adams-weka and adams-weka-lts: The Text directory source in the Weka Investigator allows the loading of directories with plain text documents as datasets.