Dear IntelMQ community
The new IntelMQ release 3.5.0 brings support for a few new data fields that many of you have requested:
If you use a database to store events, update the schema of the events table with the SQL statements shown below.
If you use deb- or rpm-packages, you have two new packages available: intelmq-contrib and intelmq-autostart. The first contains stuff from the repositories' contrib/ folder: some eventdb scripts, an example extension package, the feeds config generator, malware name mapping scripts, and systemd scripts. The autostart package contains systemd units to start IntelMQ at boot time and then run it periodically.
The full list of changes is very long, below you can read the full list of changes.
Thanks to all who contributed to this version, whether through user feedback or code.
This version contains code contributions from Edvard Rejthar, Kamil Mańkowski, Karl-Johan Karlsson, Lukas Heindl, Mikk Margus Möll, Nakul Rajpal, Sebastian Wagner, Tim de Boer.
The 3.5.0 version is available at GitHub, PyPI and in the package repositories:
https://github.com/certtools/intelmq/releases/tag/3.5.0
https://pypi.org/project/intelmq/
https://docs.intelmq.org/latest/admin/installation/linux-packages/
The complete change log:
## Requirements
Python `>=3.9` is now required, which is available on all platforms supported by IntelMQ.
## Data Format
To save new fields from IntelMQ Data Format in existing PostgreSQL instances, the following schema update is necessary:
```sql
CREATE TYPE severity_enum AS ENUM (
'critical',
'high',
'medium',
'low',
'info',
'undefined'
);
ALTER TABLE events ADD "product.full_name" text;
ALTER TABLE events ADD "product.name" text;
ALTER TABLE events ADD "product.vendor" text;
ALTER TABLE events ADD "product.version" text;
ALTER TABLE events ADD "product.vulnerabilities" text;
ALTER TABLE events ADD severity varchar(10);
ALTER TABLE events ADD "constituency" text;
UPDATE events SET severity = (extra ->> 'severity')::severity_enum;
```
Optionally remove the severity field from the extra fields in existing entries:
```sql
UPDATE events SET extra = extra - 'severity';
```
To switch to the more efficient data type `jsonb` instead of `json`, use the following SQL statement. Data is preserved. JSONB also has more query and data manipulation features than plain JSON.
```sql
ALTER TABLE events
ALTER COLUMN "extra" SET DATA TYPE jsonb;
```
## Configuration
- New parameter `stop_retry_limit` (PR#2598 by Lukas Heindl).
## Core
- Drop support for Python 3.8 (fixes #2616, PR#2617 by Sebastian Wagner).
- `intelmq.lib.splitreports`: Handle bot parameter `chunk_size` values empty string, due to missing parameter typing checks (PR#2604 by Sebastian Wagner).
- `intelmq.lib.mixins.sql` Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
- New parameter `stop_retry_limit` to gracefully handle stopping bots which take longer to shutdown (PR#2598 by Lukas Heindl, fixes #2595).
- `intelmq.lib.datatypes`: Remove unneeded Dict39 alias (PR#2639 by Nakul Rajpal, fixes #2635)
- `intelmq.lib.mixins.http`: Only set HTTP header 'Authorization' if username or password are set and are not both empty string as they are by default in the Manager (fixes #2590, PR#2634 by Sebastian Wagner).
- `intelmq.lib.message.Message.from_dict`: Do not modify the dict parameter by adding the `__type` field and raise an error when type is not determinable (PR#2545 by Sebastian Wagner).
- `intelmq.lib.mixins.cache.CacheMixin` was extended to support temporary storing messages in a cache queue
(PR#2509 by Kamil Mankowski).
## Data Format
- Implementing [IEP009](https://github.com/certtools/ieps/tree/main/009) introducing fields to identify products and vulnerabilities: `product.full_name`, `product.name`, `product.vendor`,
`product.version`, `product.vulnerabilities`. To store in existing PostgreSQL instances, a schema update is necessary, see first section.
- added `severity` field to help with triaging received events (PR#2575 by Kamil Mańkowski). To allow saving the field in PostgreSQL database in existing installations, the schema update shown in the first section is necessary.
- Implementing [IEP008](https://github.com/certtools/ieps/tree/main/008) introducing the `constituency` field for easier identification in multi-constituency setups. (PR#2573 by Kamil Mańkowski). To use in current PostgreSQL installations, a schema update is necessary, see first section.
## Bots
## Collectors
- `intelmq.bots.collectors.mail.collector_mail_attach`: Decrypt GPG attachments (PR#2623 by Edvard Rejthar).
- `intelmq.bots.collectors.mail.collector_mail_attach`: Allow empty attachments (PR#2647 by Edvard Rejthar).
- `intelmq.bots.collectors.shodan.collector_alert`: Added a new collector to query the Shodan Alert API (PR#2618 by Sebastian Wagner and Malawi CERT).
- Remove `intelmq.bots.collectors.blueliv` as it uses an unmaintained library, does not work any more and breaks other CI tests (fixes #2593, PR#2632 by Sebastian Wagner).
## Parsers
- `intelmq.bots.parsers.cymru.parser_cap_program`: Add mapping for TOR and ipv6-icmp protocol (PR#2621 by Mikk Margus Möll).
- Remove `intelmq.bots.parsers.blueliv` as it is obsolete with the removed collector (PR#2632 by Sebastian Wagner).
- `intelmq.bots.parser.json.parser`:
- Support data containing lists of JSON Events (PR#2545 by Tim de Boer).
- Add default `classification.type` with value `undetermined` if input data has now classification itself (PR#2545 by Sebastian Wagner).
## Experts
- `intelmq.bots.experts.asn_lookup.expert`:
- Print URLs to stdout only in verbose mode (PR#2591 by Sebastian Wagner).
- Check for database file existence and writability (fixes #2566).
- Use database path matching to installation type (PR#2606 by Sebastian Wagner).
- `intelmq.bots.experts.fake.expert`:
- Use database path matching to installation type (PR#2606 by Sebastian Wagner).
- Add new mode `random_single_value` (PR#2601 by Sebastian Wagner).
- `intelmq.bots.experts.sieve.expert`: Test for textX dependency in self-check (PR#2605 by Sebastian Wagner).
- `intelmq.bots.experts.trusted_introducer_lookup.expert`: Change to new TI database URL (fixes #2620, PR#2633 by Sebastian Wagner).
## Outputs
- `intelmq.bots.outputs.smtp_batch.output`:
- Add new parameter `additional_grouping_keys` for an enhanced email batching feature.
- Add new parameter `templating` for additional template variables.
- Add new parameter `allowed_fieldnames` for csv field specification.
- Add new parameter `fieldnames_translation` for naming csv headers (PR#2610 by Lukas Heindl, fixes #2586).
- `intelmq.bots.outputs.sql.output`: Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
## Documentation
- Fix and refresh links to mailing lists (PR#2609 by Kamil Mańkowski)
- `Aggregate Bot`: Add illustration graphics (PR#2612 by Sebastian Wagner).
## Packaging
- Replace `/opt/intelmq` example paths in bots with variable `VAR_STATE_PATH` for correct paths in LSB-path setups like with packages (PR#2587 by Sebastian Wagner).
- New deb-package `intelmq-contrib` with all `contrib/` scripts and documentation (PR#2614 by Sebastian Wagner).
- New deb-package `intelmq-autostart` containing systemd services and timers to start all enabled IntelMQ bots at boot and periodically (PR#2638 by Sebastian Wagner).
## Tests
- `intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge`: Skip on all Python versions when running on CI (PR#2602 by Sebastian Wagner).
- `.github/workflows/codespell.yml`, `debian-package.yml`, `regexploit.yml`: Upgrade to `ubuntu-latest` runners (PR#2602 by Sebastian Wagner).
- `intelmq.test.test_conf`: With changed behaviour in ruamel.yaml on line wrapping since version 0.18.13, only test the parsabilty of `runtime.yaml` (PR#2619 by Sebastian Wagner).
- `intelmq.test.BotTestCase.test_static_bot_check_method`: Remove debugging stub raising for all non-empty checks (PR#2622 by Sebastian Wagner).
## Tools
- `intelmq.bin.intelmq_psql_initdb`: Use `JSONB` type by default, Postgres supports it since version 9 (PR#2597 by Sebastian Wagner).
- `intelmq.bin.rewrite_config_files`: Removed obsolete JSON configuration file rewriter (PR#2613 by Sebastian Wagner).
- `intelmq/lib/bot_debugger.py`: Fix overwriting the runtime logging level by command line parameter (PR#2603 by Sebastian Wagner, fixes #2563).
- `intelmq.bin.intelmqctl`: Fix bot log level filtering (PR#2607 by Sebastian Wagner, fixes #2596).
## Contrib
- Bash Completion: Adapt to YAML-style runtime configuration (PR#2642 by Sebastian Wagner, fixes #2094).
- Remove `prettyprint` script, use `jq` instead (PR#2551 by Sebastian Wagner).
## Known issues
This is short list of the most important known issues. The full list can be retrieved from [GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+label%3Abug).
- stomp.py 8.2.0+ breaks the version check in stomp bots (#2600).
- Traceback when calling intelmqdump without write access to the log file (#2529).
- pyyaml PendingDeprecationWarning: you should no longer specify 'unsafe' -> test failure (#2489).
- `intelmq.parsers.html_table` may not process invalid URLs in patched Python version due to changes in `urllib` (#2382).
- Breaking changes in 'rt' 3.0 library (#2367).
- Type error with SQL output bot's `prepare_values` returning list instead of tuple (#2255).
- `intelmq_psql_initdb` does not work for SQLite (#2202).
- intelmqsetup: should install a default state file (#2175).
- Misp Expert - Crash if misp event already exist (#2170).
- Spamhaus CERT parser uses wrong field (#2165).
- Custom headers ignored in HTTPCollectorBot (#2150).
- intelmqctl log: parsing syslog does not work (#2097).
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
- Corrupt dump files when interrupted during writing (#870).
-- Institute for Common Good Technology gemeinnütziger Kulturverein - nonprofit cultural society https://commongoodtechnology.org/ ZVR 1510673578