Dear IntelMQ community
The new IntelMQ release 3.5.0 brings support for a few new data fields that many of you have requested:
* severity * product.full_name, product.name, product.vendor, product.version, product.vulnerabilities (IEP009: IntelMQ Data Format: Describe product and mark vulnerabilities https://github.com/certtools/ieps/tree/main/009) * constituency (IEP008: IntelMQ Data Format: Constituency field https://github.com/certtools/ieps/tree/main/008)
If you use a database to store events, update the schema of theevents table with the SQL statements shown below.
If you use deb- or rpm-packages, you have two new packages available: intelmq-contrib and intelmq-autostart. The first contains stufffrom the repositories' contrib/ folder: some eventdb scripts, an example extension package, the feeds config generator, malware name mapping scripts, and systemd scripts.The autostart package contains systemd units to start IntelMQ at boot time and then run it periodically.
The full list of changes is very long, belowyou can read the full list of changes.
Thanks to all who contributed to this version, whether through user feedback or code.
This version contains code contributions from Edvard Rejthar, Kamil Mańkowski, Karl-Johan Karlsson, Lukas Heindl, Mikk Margus Möll, Nakul Rajpal, Sebastian Wagner, Timde Boer.
The 3.5.0 version is available at GitHub, PyPI andin the package repositories:
https://github.com/certtools/intelmq/releases/tag/3.5.0
https://pypi.org/project/intelmq/
https://docs.intelmq.org/latest/admin/installation/linux-packages/
The complete change log:
## Requirements
Python `>=3.9` is now required, which is available on all platforms supported by IntelMQ.
## Data Format
To save new fields from IntelMQ Data Format in existing PostgreSQL instances, the following schema update is necessary:
```sql
CREATE TYPE severity_enum AS ENUM (
'critical',
'high',
'medium',
'low',
'info',
'undefined'
);
ALTER TABLE events ADD "product.full_name" text;
ALTER TABLE events ADD "product.name" text;
ALTER TABLE events ADD "product.vendor" text;
ALTER TABLE events ADD "product.version" text;
ALTER TABLE events ADD "product.vulnerabilities" text;
ALTER TABLE events ADD severity varchar(10);
ALTER TABLE events ADD "constituency" text;
UPDATE events SET severity = (extra ->> 'severity')::severity_enum;
```
Optionally remove the severity field from the extra fields in existing entries:
```sql
UPDATE events SET extra = extra - 'severity';
```
To switch to the more efficient data type `jsonb` instead of `json`, use the following SQL statement. Data is preserved. JSONB also has more query and data manipulation features than plain JSON.
```sql
ALTER TABLE events
ALTER COLUMN "extra" SET DATA TYPE jsonb;
```
## Configuration
- New parameter `stop_retry_limit` (PR#2598 by Lukas Heindl).
## Core
- Drop support for Python 3.8 (fixes #2616, PR#2617 by Sebastian Wagner).
- `intelmq.lib.splitreports`: Handle bot parameter `chunk_size` values empty string, due to missing parameter typing checks (PR#2604 by Sebastian Wagner).
- `intelmq.lib.mixins.sql` Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
- New parameter `stop_retry_limit` to gracefully handle stopping bots which take longer to shutdown(PR#2598 by Lukas Heindl, fixes #2595).
- `intelmq.lib.datatypes`: Remove unneeded Dict39 alias (PR#2639 by Nakul Rajpal, fixes #2635)
- `intelmq.lib.mixins.http`: Only set HTTP header 'Authorization' if username or password are set and are not both empty stringas they are by default in the Manager (fixes #2590, PR#2634 by Sebastian Wagner).
- `intelmq.lib.message.Message.from_dict`: Do not modify the dict parameter by adding the `__type` field and raise an error when type is not determinable (PR#2545 by Sebastian Wagner).
- `intelmq.lib.mixins.cache.CacheMixin` was extended to support temporarystoring messages in a cache queue
(PR#2509 by Kamil Mankowski).
## Data Format
- Implementing [IEP009](https://github.com/certtools/ieps/tree/main/009) introducingfields to identify products and vulnerabilities: `product.full_name`, `product.name`, `product.vendor`,
`product.version`, `product.vulnerabilities`. To store in existing PostgreSQL instances, a schema update is necessary, see firstsection.
- added `severity` field to help with triaging received events (PR#2575 by Kamil Mańkowski). To allow saving the field in PostgreSQLdatabase in existing installations, the schema update shown in the first section is necessary.
- Implementing [IEP008](https://github.com/certtools/ieps/tree/main/008) introducingthe `constituency` field for easier identification in multi-constituency setups. (PR#2573 by Kamil Mańkowski). To use in current PostgreSQL installations, a schema update is necessary, see firstsection.
## Bots
## Collectors
- `intelmq.bots.collectors.mail.collector_mail_attach`: Decrypt GPG attachments (PR#2623 by Edvard Rejthar).
- `intelmq.bots.collectors.mail.collector_mail_attach`: Allow empty attachments (PR#2647 by Edvard Rejthar).
- `intelmq.bots.collectors.shodan.collector_alert`: Added a new collector to query the Shodan Alert API (PR#2618 by Sebastian Wagner and Malawi CERT).
- Remove `intelmq.bots.collectors.blueliv` as it uses an unmaintained library, does not work any moreandbreaks other CI tests (fixes #2593, PR#2632 by Sebastian Wagner).
## Parsers
- `intelmq.bots.parsers.cymru.parser_cap_program`: Add mapping for TOR and ipv6-icmpprotocol (PR#2621 by Mikk Margus Möll).
- Remove `intelmq.bots.parsers.blueliv` as it is obsolete with the removed collector (PR#2632 by Sebastian Wagner).
- `intelmq.bots.parser.json.parser`:
- Support data containing lists of JSON Events (PR#2545 by Tim de Boer).
- Add default `classification.type` with value `undetermined` if input data has nowclassification itself (PR#2545 by Sebastian Wagner).
## Experts
- `intelmq.bots.experts.asn_lookup.expert`:
- Print URLs to stdout only in verbose mode (PR#2591 by Sebastian Wagner).
- Check for database file existence and writability (fixes #2566).
- Use database path matching to installation type (PR#2606 by Sebastian Wagner).
- `intelmq.bots.experts.fake.expert`:
- Use database path matching to installation type (PR#2606 by Sebastian Wagner).
- Add new mode `random_single_value` (PR#2601 by Sebastian Wagner).
- `intelmq.bots.experts.sieve.expert`: Test for textX dependency in self-check (PR#2605 by Sebastian Wagner).
- `intelmq.bots.experts.trusted_introducer_lookup.expert`: Change to new TI database URL (fixes #2620, PR#2633 by Sebastian Wagner).
## Outputs
- `intelmq.bots.outputs.smtp_batch.output`:
- Add newparameter `additional_grouping_keys` for an enhanced email batching feature.
- Add newparameter `templating` for additional template variables.
- Add new parameter `allowed_fieldnames` for csvfield specification.
- Add new parameter `fieldnames_translation` for naming csvheaders (PR#2610 by Lukas Heindl, fixes #2586).
- `intelmq.bots.outputs.sql.output`: Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
## Documentation
- Fix and refresh links to mailing lists (PR#2609 by Kamil Mańkowski)
- `Aggregate Bot`: Add illustration graphics (PR#2612 by Sebastian Wagner).
## Packaging
- Replace `/opt/intelmq` example paths in bots with variable `VAR_STATE_PATH` for correct paths in LSB-path setups likewith packages (PR#2587 by Sebastian Wagner).
- New deb-package `intelmq-contrib` with all `contrib/` scripts and documentation (PR#2614 by Sebastian Wagner).
- New deb-package `intelmq-autostart` containing systemd services and timers to start all enabled IntelMQ bots at boot and periodically (PR#2638 by Sebastian Wagner).
## Tests
- `intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge`: Skip on all Python versions when running on CI (PR#2602 by Sebastian Wagner).
- `.github/workflows/codespell.yml`, `debian-package.yml`, `regexploit.yml`: Upgrade to `ubuntu-latest` runners (PR#2602 by Sebastian Wagner).
- `intelmq.test.test_conf`: With changed behaviourin ruamel.yaml on line wrapping since version 0.18.13, only test the parsabiltyof `runtime.yaml` (PR#2619 by Sebastian Wagner).
- `intelmq.test.BotTestCase.test_static_bot_check_method`: Remove debugging stub raising for all non-empty checks (PR#2622 by Sebastian Wagner).
## Tools
- `intelmq.bin.intelmq_psql_initdb`: Use `JSONB` type by default, Postgres supports it since version 9 (PR#2597 by Sebastian Wagner).
- `intelmq.bin.rewrite_config_files`: Removed obsolete JSON configuration file rewriter (PR#2613 by Sebastian Wagner).
- `intelmq/lib/bot_debugger.py`: Fix overwriting the runtime logging level by command line parameter (PR#2603 by Sebastian Wagner, fixes #2563).
- `intelmq.bin.intelmqctl`: Fix bot log level filtering (PR#2607 by Sebastian Wagner, fixes #2596).
## Contrib
- Bash Completion: Adapt to YAML-style runtime configuration (PR#2642 by Sebastian Wagner, fixes #2094).
- Remove `prettyprint` script, use `jq` instead (PR#2551 by Sebastian Wagner).
## Known issues
Thisis shortlist of the most important known issues. The fulllist can be retrievedfrom [GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+label...).
- stomp.py 8.2.0+ breaks the version check in stomp bots (#2600).
- Traceback when calling intelmqdump without write access to the log file (#2529).
- pyyaml PendingDeprecationWarning: you should no longer specify 'unsafe' -> test failure (#2489).
- `intelmq.parsers.html_table` may not process invalid URLs in patched Python version due to changes in `urllib` (#2382).
- Breaking changes in 'rt' 3.0 library (#2367).
- Type error with SQL output bot's `prepare_values` returning list instead of tuple(#2255).
- `intelmq_psql_initdb` does not work for SQLite (#2202).
- intelmqsetup: should install a default state file (#2175).
- Misp Expert - Crash if misp event already exist(#2170).
- Spamhaus CERT parser uses wrongfield (#2165).
- Custom headers ignored in HTTPCollectorBot (#2150).
- intelmqctl log: parsing syslog does not work (#2097).
- Bots started with IntelMQ-API/Manager stop when the webserveris restarted (#952).
- Corrupt dump files when interrupted during writing (#870).