Release 2.4.1 - 06/14/2022 * Implement bulk upload in the OpenSearch emitter (TIKA-3791). * Implement tika-server client via pipes mode (TIKA-3790). * Custom embedded parsers and EmbeddedDocumentHandlers can now add metadata to the container file's metadata (TIKA-3789). * Record embedded file exceptions in the container file's metadata (TIKA-3788). * Allow continuation of parsing after write limit has been reached (TIKA-3787). * Allow pass-through of 'Content-Length' header to metadata in TikaResource (TIKA-3786). * Add embedded depth to profiles tables in tika-eval (TIKA-3775). * Add stop() method to TikaServerCli so that it can be run with Apache Commons Daemon (TIKA-1570). * Fixed bug in ordering of Parsers during service loading (TIKA-3750). * Users can expand system properties from the forking process into forked tika-server processes (TIKA-3748). * Fix a few files being wrongly detected as EML (TIKA-3771). * Fix ignoreCharsets param of Icu4jEncodingDetector (TIKA-3774). Release 2.4.0 - 04/23/2022 * NOTE: To save on resources, we no longer include the deeplearning4j dependencies in the tika-dl jar. The dependencies for the tika-dl package must be provided by users. See: https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-ml/tika-dl/pom.xml for the dependencies that must be provided at run-time (TIKA-3676). * NOTE: Added prefix "dwg-custom:" to DWG custom metadata properties (TIKA-3731). * Add initial, BETA-grade TLS encryption option for tika-server; configuration may change in future releases (TIKA-3719). * Allow specification of fetcherName and fetchKey via query parameters in request URI in tika-server (TIKA-3714). * Add basic parsers for WARC and WACZ in tika-parsers-standard (TIKA-3697). * Add MetadataWriteFilter capability to improve memory profile in Metadata objects (TIKA-3695). * Allow configurability of the ContentHandlerDecorator used by the AutoDetectParser (TIKA-3723). * Allow configurability of the EmbeddedDocumentExtractor used by the AutoDetectParser (TIKA-3711). * Add detection for Frictionless Data packages and WACZ (TIKA-3696). * Add detection for DGN files with gratitude and credit to Steven Frew's tika-dgn-detector (TIKA-3721). * Add parser for metadata from DGN 8 files via Dan Coldrick (TIKA-3721). * Add a fetcher and emitter for Azure blob storage (TIKA-3707). * Add detection for files encrypted by Microsoft's Rights Management Service (TIKA-3666). * Fixed regression in 2.3.0 that led to more embedded filenames than appropriate being written to the content (TIKA-3711). * tika-server now clones forking process' environment variables into forked process (TIKA-3715). * Add an optional /eval endpoint for tika-eval profile or compare capabilities in tika-server (TIKA-3689). * Add a Parsed-By-Full-Set metadata item to record all parsers that processed a file (TIKA-3716). * Add metadata filters for Optimaize and OpenNLP language detectors (TIKA-3717). * Upgrade to PDFBox 2.0.26 (TIKA-3726). * Upgrade deeplearning4j to 1.0.0-M2 (TIKA-3458 and PR#527). * Various dependency upgrades, including POI, dl4j, gson, jackson, twelvemonkeys, log4j2 and others (TIKA-3675 and many PRs from dependabot). * Switch cipher from ECB to GCM in HttpClientFactory (TIKA-3724). Release 2.3.0 - 02/02/2022 * Upgrade to Apache POI 5.2.0. This is the first upgrade to POI 5.x and represents a major refactoring. Users may experience significantly more logging (TIKA-3164). * Upgrade to log4j2 2.17.1 (TIKA-3638). * Improve consistency in reporting package-entry divs across all parsers for embedded files (TIKA-3644). This leads to some more text (embedded file names) in files with many embedded attachments. * Improve configuration of maps as params for parsers in TikaConfig (TIKA-3645). * Improve identification of iWorks 13 files and add parsing for thumbnails, some metadata and attachments (TIKA-3634). Skip handling of .iwa files, which are not yet supported. * Limit the default in-memory processing (maxMainMemoryBytes) in the PDFParser to 512MB as in the 1.x branch (TIKA-3642). * Added IDML Parser from 1.x series to 2.x series (TIKA-3188). * Extract annotation types and subtypes for PDFs into metadata (TIKA-3653). * Add metadata value for PDFs that contain 3D annotations (TIKA-3653). * Add parser for Translation Memory eXchange (TMX) files (TIKA-3660). * Add Bill of Materials (Maven BOM) for centralized module version management (TIKA-3367). Release 2.2.1 - 12/19/2021 * Fix multithreading bug for ooxml files (TIKA-3627). * Upgrade log4j to 2.17.0 (TIKA-3625). * Upgrade to PDFBox 2.0.25 (TIKA-3622) * Fix bug that prevented metadata keys in the UnpackerResource in tika-server (TIKA-3624). * Upgrade log4j to 2.16.0 (TIKA-3623) Release 2.2.0 - 12/13/2021 * Add support for OneNote files downloaded from O365 (TIKA-3446). * Fix logic bug in PipesServer that prevented concatenation of content from attachments (TIKA-3609). * Improve extraction of embedded files from MSOffice files created by non-Microsoft tools (TIKA-3526). * Added back ability to ignore load errors in TikaConfig (TIKA-3575). * Make SecureContentHandler and other parameters configurable in AutoDetectParser programmatically and via tika-config.xml (TIKA-3594). * Fix default logging in tika-app in batch mode (TIKA-3589). * Fix bug that prevented specifying a config with the long --config= option in tika-app in batch mode (TIKA-3589). * Fix thread starvation after numerous restarts in PipesClient (TIKA-3588). * Fix race condition when starting multiple forked servers on multiple ports (TIKA-3586). * Add timeout per task to be configured via headers for tika-server's legacy endpoints /tika and /rmeta. Note that this timeout greater than taskTimeoutMillis (TIKA-3582). * Add metadata item for whether or not a PDF has a collection/ is a Portfolio PDF (TIKA-3579). * Add detection of ESRI Layer files (TIKA-3570). * Add detection of JPEG XL, MARC, ICC profiles, NES-ROM file types (TIKA-3562 and TIKA-3563) * Remove duplicate "subject" metadata keys that were intended for backwards compatibility within 1.x only (TIKA-3564). * Fix Open Office mime types to be subclasses of application/zip and no longer require OPCPackageDetector-last ordering of zip detectors (TIKA-3556). * Improve robustness and features of the httpfetcher (TIKA-3543) * Add optional fetch ranges to FetchEmitTuple to allow range fetching from, e.g. http or s3 (TIKA-3542). * Exclude dependencies on jsoup and ehcache in ucar grib/cdm (TIKA-3003). Release 2.1.0 - 08/18/2021 MAJOR CHANGES in 2.1.0: * Improved packaging for tika-parsers-extended. Use the tika-parser-scientific-package and tika-parser-sqlite3-package artifacts if you want fat jars with dependencies. (TIKA-3510) * Tika app writes UTF-8 when an encoding is not specified; the legacy behavior was UTF-8 on Mac OS, but System default on other OSs (TIKA-3515). * Change the default rendering strategy for PDFs from NO_TEXT to ALL (TIKA-3520). Other changes: * Fixed bug that pointed to the wrong tessdata directory if the user specified a tesseract path but not also a tessdata path (TIKA-3518). * Fixed bug in Icu4j's encoding detector where it would return non-standard names for charsets, e.g. IBM424_rtl is now returned as IBM424 (TIKA-3516). * Add a simple UrlFetcher in tika-core as a basic alternative to tika-fetcher-http (TIKA-3527). * Add tika-pipes support for Google Cloud Storage (TIKA-3524). * Fix markup ordering errors in xhtml output for ODT files (TIKA-2242). * Fix serialization of embedded docs in OpenSearch emitter and fix embedded documents not being indexed in some use cases in the Solr emitter (TIKA-3490). * Add pipesClientId system property to PipesServer so that each forked process can log to its own logger (TIKA-3480). * Add DateNormalizingMetadataFilter let users ensure that all dates emitted to Solr/OpenSearch are in UTC. Users can configure which timezone they'd like to use in cases where the file format does not store a timezone (TIKA-3496). * Breaking change in the Solr and OpenSearch emitters. To achieve the SKIP or CONCATENATE attachment strategy, modify the parseMode in the pipesiterators or in the FetchEmitTuple (TIKA-3494). Release 2.0.0 - 07/07/2021 * Cleanup of fetcher integration with tika-server. * Update dependencies. Release 2.0.0-BETA - 05/19/2021 * Refactor pipes module for resilience * Add transcribe capability (TIKA-94). Release 2.0.0-ALPHA - 01/13/2021 BREAKING CHANGES in 2.0.0 * General * OCR is now triggered automatically for PDFs if tesseract is on the user's path see (https://cwiki.apache.org/confluence/display/TIKA/TikaOCR#TikaOCR-disable-ocr) for how to disable OCR. * We upgraded from log4j to log4j2 in tika-app, tika-server and anywhere else we used to use log4j. * By default, when rendering a page for OCR, the PDFParser does not render glyphs/text. * Removed deprecated Metadata keys/properties (TIKA-1974). * Removed deprecated PDFPreflightParser (TIKA-3437). * Removed dangerous calls to read an inputstream or convert to bytes without specifying a charset * Parsers can be configured via tika-config.xml on instantiation. We have moved away from configuration via .properties files because of confusion among users. This affects the PDFParser, TesseractOCRParser and the StringsParser. * Changed namespaces of translator implementations (o.a.t.language.translate.impl) to avoid split-package with tika-core * tika-parsers * The parser modules have been broken into three main modules: tika-parsers-standard, tika-parsers-extended and tika-parsers-ml. Users may now need to add tika-parsers-extended's tika-parser-scientific-module or tika-parser-sqlite3-module to tika-app and tika-server to include parsers that used to be included by default (for example: envi, gdal, grib, isatab, netcdf, sqlite3). * PDFParser -- a) see above on OCR. b) This parser no longer warns if the jpeg2000 dependency is not included. Tika now relies on PDFBox to log an error if a jpeg2000 image should be processed but can't because the required external dependency is not available. See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io for the non-ASF-2.0-compatible jpeg2000 library. * CompressorParser -- users must add the com.github.luben:zstd-jni dependency to the classpath to process zstd files. This is an optional library that is no longer bundled in tika-parsers-standard-package because it contains native libs. * ChmParser was moved to org.apache.tika.parser.microsoft.chm * RTFParser was moved to org.apache.tika.parser.microsoft.rtf * We are now using non-shaded versions of xmpcore with namespaces com.adobe.internal.* vs com.adobe.*. * tika-app * See above on default inclusion of only tika-parsers-standard. * tika-server * tika-server now by default forks a process to isolate the parsing in the forked process (this was called the -spawnChild option in tika-1.x). Clients must now expect that tika-server will restart on OOM, timeouts, crashes or after parsing a large number of files. When this happens tika-server will restand and not receive connections for brief periods. The less robust, legacy behavior of not forking a process is available with "-noFork"= * Most of tika-server's legacy configuration via the commandline has been moved into configuration via a tika-config.xml file. * tika-server's "enableFileUrl" has been removed in favor of a FileSystemFetcher. * tika-server's /metadata endpoint requires tika-server-standard to write XMP/rdf output. This output is not available in tika-server-core. * In tika-server, for those parsers that can be configured per parse via a config object passed in through the ParseContext, the config object will only update those fields that the user has modified. The config object will no longer fully reset all settings to the default settings per parse. This has a more intuitive "update the base/configured settings" with what has been changed in the config object. * tika-eval * tika-eval's default profile and comparison reports no longer include tag reports. Users can get the report configs that include tags (*-tags.xml): https://github.com/apache/tika/tree/main/tika-eval/tika-eval-app/src/main/resources Release 1.27 - 06/30/2021 * Migrate MP4 parsing to Drew Noakes' metadata-extractor (TIKA-3459). To revert to legacy parser turn off NoakesMP4Parser and turn on MP4Parser via tika-config.xml. * Prevent rare infinite loop in tika-server's -spawnChild mode when restart fails because of failure to bind to the port (TIKA-3441). * Improve likelihood that tesseract will not be orphaned on jvm restart in tika-server (TIKA-3441). * Deprecate experimental PDFPreflightParser (TIKA-3437). * Apply encoding detection to zip entry names via Ryan421 (TIKA-3374). * Add json output for /tika endpoint in tika-server (TIKA-3352). * Tika's PDFParser should use the underlying file if one is passed in via a TikaInputStream (TIKA-3350) Release 1.26 - 03/24/2021 * Fix thread safety bug in OpenOffice parser (TIKA-3334). * The "writeLimit" header now pertains to the combined characters written per container document (and embedded documents) in the /rmeta endpoint in tika-server (TIKA-3325); it no longer functions only per container or embedded document. * Extract more embedded files in PDFs by recursively processing the embedded file tree (TIKA-3332). * Allow for case insensitive headers for configuration of the PDFParser and the TesseractOCRParser in tika-server via Subhajit Das (TIKA-3320). * Improve detection and parsing of XPS files (TIKA-3316). * General dependency upgrades (TIKA-3244). * Great optimization in ForkParser (TIKA-3237). * Fix parsing of emails attached to other emails in PST files (TIKA-3004). * MP3 parser should output the xmpDM:duration metadata as seconds not milliseconds, consistent with the other Audio and Video parsers (TIKA-3318). * MP4 parser check if any of the Compatible Brands match when identifying the subtype (TIKA-3310). Release 1.25 - 11/25/2020 * Fix inconsistent license in xmpcore (TIKA-3204). * General upgrades including some dependencies with recently found security vulnerabilities (TIKA-3119). * Add detection and a parser for flat ODF files (TIKA-3159). * Add extraction of macros from ODF files (TIKA-3161). * Add mime detection for hprof and hprof text files (TIKA-3144). * Add TextSignature and TextProfileSignature to tika-eval (TIKA-3145 and TIKA-3146) * Create a metadata filter to trigger tika-eval stats post parsing (TIKA-3140) * Add a configurable metadata-filter for the RecursiveParserWrapper (TIKA-3137) * Parameterize writeLimit and maxEmbeddedResources for RecursiveParserWrapper in tika-server (TIKA-3133) * Add status endpoint to tika-server (TIKA-3129). * Remove whitelist/blacklist terminology (TIKA-3120) * Add detection for parquet files (TIKA-3115). * Add detection and parsing for bplist (TIKA-3104). * Enable metadata value filtering for RecursiveParserWrapper (TIKA-3137) * Add a basic parser for plist files based on com.googlecode.plist:dd-plist (TIKA-3104). * Read hyperlinked images from ODT files (TIKA-3156). * Updated GrobidRESTParser to use new API location (TIKA-3191). * Add FileProfiler to tika-eval (TIKA-3216). * Add status endpoint to tika-server (TIKA-3129). * Improved handling of zip files with STORED entries with data descriptor (TIKA-3196). * Add parsers for XLZ, IDML and MIF (TIKA-2976, TIKA-3188 and TIKA-3189). * Add the beginnings of a format-aware fuzzing module (TIKA-3083). * Add wrapper for Linux 'file' command for mime detection (TIKA-3215). * Added ability to skip parsing of embedded files in Tika Server (TIKA-3227). Release 1.24.1 - 4/17/2020 * Allow gzip compression of input and output streams for tika-server (TIKA-3073). Release 1.24 - 3/11/2019 * Add scripts to run tika-server as a service via Eric Pugh, and add these scripts and jar as a new artifact in the release (TIKA-3010). * Upgrade Drew Noakes' metadata-extractor (TIKA-2952). * Enable optional extraction of structural tags in PDFs (alpha-grade) (TIKA-3026). * Tika app's --extract mode now outputs to STDOUT (TIKA-3035). * Add an optional Preflight parser for PDFs (TIKA-3055). * Improve detection of some zip-based formats (TIKA-3057). * Upgrade metadata-extractor to 2.13.0 (TIKA-2952). * Upgrade to POI 4.1.2 (TIKA-3047). * Extract XMP from PSD files (TIKA-3050). * Added XMLProfiler as an optional parser to profile XFA and XMP in PDFs (TIKA-3045). * Extract inline images that rely on the DCT filter from PDFs (TIKA-3041). * Upgrade to PDFBox 2.0.19 (TIKA-3033). * Fix bug in ASM parser configuration (TIKA-2992). * Upgrade to java-libpst 0.9.3 (TIKA-2546). * Fixed XLIFF12Parser failures with ToXMLHandler (TIKA-3014). Release 1.23 - 12/02/2019 * NOTE: The PDFParser now relies on OCRDPI to render page images when users configure OCR on rendered page images. This will have the effect of increasing rendered image size (TIKA-2624). * NOTE: tika-server no longer returns 415 for file types for which there is no parser. * Fix bug in AUTO OCR strategy in the PDFParser (TIKA-3002). * Fix incorrect height and width metadata extraction from JPEG images (TIKA-2630). * Upgrade to POI 4.1.1 (TIKA-2851). * Upgrade to PDFBox 2.0.17 (TIKA-2951). * Ensure that the PDFParser respects custom configuration of Tesseract from tika-config.xml via Eric Pugh (TIKA-2970). * Add parser for XLIFF v1.2 files (TIKA-2975). * Add mime type detection support for WebAssembly (TIKA-2894), HEIF / HEIC images (TIKA-2942), Digilite FDF (TIKA-2988); and xml-root detection for XFDF (TIKA-2990) and XDP (TIKA-2989). * Add an XLZ Parser (TIKA-2976). * Fix deadlock with ForkParser when InputStream throws IOException (TIKA-2892). Release 1.22 - 07/29/2019 * NOTE: tika-server no longer hard-codes the HtmlParser to handle XML files (TIKA-2910). Users must now configure that behavior via a tika-config.xml file. * NOTE: Known regression: PDFBOX-4587 -- PDF passwords with codepoints between 0xF000 and 0XF0000 will cause an exception. * Add parser for HWP v5 files via SooMyung Lee (soomyung) and JinSup Kim (ddoleye) (TIKA-2909). * Fix order of closing streams to avoid "Failed to close temporary resource" exception in TesseractOCRParser (TIKA-2908). * Improve AutoDetectReader performance by caching encoding detector (TIKA-1568). * Prevent RTFParser from outputting illegal tag combinations (TIKA-2889). * Fix RereadableInputStream to release all resources (TIKA-2903). * Implement custom language identifier in the tika-eval module based on OpenNLP's language detector; add 18 languages and add common words lists for all 121 languages (TIKA-2790). * Fix NPE in MimeTypesReader.releaseParser() via Eamonn Saunders (TIKA-2896). * Fix RTFParser to extract more content (TIKA-2883). * Add clientSubmitTime to the metadata extracted from PST files (TIKA-2898). * Improve StreamingZipContainerDetector for xltx, xltm and several other file formats (TIKA-2886). Release 1.21 - 05/14/2019 * Add optional AUTO mode to OCR'ing of PDFs. If tesseract is installed and on the path, and this option is selected programmatically or via TikaConfig(), the PDFParser will use heuristics to decide whether or not to run OCR per page on PDFs. (TIKA-2749) * The ZipContainerDetector's default behavior was changed to run streaming detection up to its markLimit. Users can get the legacy behavior (spool-to-file/rely-on-underlying-file-in-TikaInputStream) by setting markLimit=-1. The POIFSContainerDetector requires an underlying file; it will try to spool the file to disk; if the file's length is > markLimit, it will not attempt detection; set markLimit to -1 for legacy behavior (TIKA-2849). * Upgrade PDFBox to 2.0.14 (TIKA-2834). * Add CSV detection and replace TXTParser with TextAndCSVParser; users can turn off CSV detection by excluding the TextAndCSVParser and adding back the TXTParser via tika-config (TIKA-2833). * Add a CSVParser. CSV detection is currently based solely on filename and/or information conveyed via Metadata (TIKA-2826). * General upgrades: asm, bouncycastle, commons-codec, commons-lang3, cxf, guava, h2, httpcomponents, jackcess, junrar, Lucene, mime4j, opennlp, parso, sqlite-jdbc (provided), zstd-jni (provided) (TIKA-2824) * Bundle xerces2 with tika-parsers (TIKA-2802). * Upgrade jaxb to 2.3.2 (TIKA-2819). * Upgrade jackson to 2.9.8 (TIKA-2717). * Update tika-eval's common tokens lists (TIKA-2822). * Handle bad tags in tika-eval more robustly (TIKA-2810). * Add reports for tags in tika-eval (TIKA-2809). * Extract text from SDT element within textboxes in .docx files (TIKA-2807). * Try to handle truncated OOXML files more robustly (TIKA-2765). Release 1.20 - 12/17/2018 * Upgrade to POI 4.0.1 (TIKA-2751). * Integrate/parameterize new angles handling in PDFBox (TIKA-2779). * Upgrade to PDFBox 2.0.13 (TIKA-2788). * Prevent content within