Streams

STREAM was added in Redis 5.0 to process log data.
In many industries, data is produced continuously. It is generated by machines rather than people, in a continuous, high-volume flow. Another trait is that existing records are not updated; new records are only appended. We call this kind of data a stream or log.
For example, semiconductor, display, and steel manufacturing lines run many sensors for temperature, humidity, pressure, vibration, inclination, illuminance, and smoke, producing data at sub-second intervals. Plants use this data to analyze product defects, and sometimes to prevent defects in advance.
semiconductor wafer defects
A closer example: monitoring many servers. You can collect CPU, Memory, Disk I/O, and Network I/O from each server, show real-time charts, and send alerts.
Redis 5.0 introduced STREAM as a data structure suited to storing and processing this kind of data.
disk vs memory, RDB table vs STREAM
The following sections walk through adding and reading stream data.

Append data: XADD

Add temperature data for sensor-1234.

XADD syntax is:

key=sensor-1234, ID=*, field=temperature, value=98.7.
Note ID=*. Passing * lets Redis generate and return the ID.
An ID has two numbers:
<millisecondsTime>-<sequenceNumber>
The first is a millisecond timestamp. The second increments 1, 2, 3, ... within the same millisecond to avoid duplicates. sequenceNumber is an 8-byte integer, so there is effectively no limit on entries in the same millisecond. ID=* uses the time of the server running Redis.
You can also specify an ID yourself.

The ID must always be greater than the previous one. Equal or smaller IDs raise an error.

Entry count: XLEN

More precisely, it returns the number of IDs.

Query data: XRANGE

XRANGE syntax is:

key=sensor-1234, start=-, end=+.
COUNT limits how many entries are returned.
start and end can be specific IDs.

To read newest entries first, use XREVRANGE.

Read data: XREAD

XREAD syntax is:

• COUNT is how many entries to read. Omit COUNT to read all matching entries.
• ID reads entries after that ID. Use 0 to start from the first entry.
• BLOCK waits for new data. In that case use the special ID $.
  This is similar to BLPOP on lists.

Delete data: XDEL, XTRIM

XDEL

XDEL syntax is:

XTRIM

If sensor-1234 has 100 entries, this deletes the oldest 90 and keeps the latest 10.

XTRIM syntax is:

The special option ~ means about.   If sensor-1234 has 1 million entries, deleting 999,990 of them takes time. While that delete runs, XADD and XREAD cannot proceed. Streams that must handle large volume should not stall like that. ~ deletes only as many entries as can be processed quickly.

Compared with other data types

LIST

Until STREAM existed, LIST was the usual Redis type for stream/log data.

• LPUSH, XADD: similar roles, different speed. LPUSH is O(1) and XADD is O(log(N)).   O(1), O(N), O(log(N))
• RPOP, XREAD: RPOP reads and removes. XREAD only reads. RPOP is O(1) and XREAD is O(log(N)). If the application fails after RPOP, there is no way to recover the entry.
  To read a LIST without removing, use LINDEX.
  To read and then remove a STREAM entry like RPOP, use XREAD and XDEL.
• LREM, XDEL: LREM is O(N) and XDEL is O(log(N)).
• LTRIM, XTRIM: LTRIM is O(N) and XTRIM is O(log(N)).
• Compare speed: LIST compares string values. STREAM compares numeric IDs, so STREAM compares faster.
• Data: LIST requires the application to parse fields and values.
  LIST: "sensor-1234;1538319053569;temperature;98.7"
  STREAM keeps fields and values separate in Redis.
• Inserting in the middle: possible with LIST, not with STREAM.

Sorted SET

Both types are ordered by score/ID. If you stored stream/log data in a ZSET, you would use timestamp as the score.

• ZADD, XADD: both O(log(N)).
• ZPOPMAX, XREAD: reading the last entry is O(log(N)) for both.
• ZREM, XDEL: both O(log(N)).
• Data: ZSET does not allow duplicate members. If the same value can arrive twice, you must add a discriminator such as a timestamp.
  Example) Score: 1538319053569, Value: "sensor-1234;1538319053569;temperature;98.7"
• Inserting in the middle: possible with ZSET, not with STREAM.

HASH

Both have fields. HGETALL does not preserve insertion order, while STREAM keeps the order fields were added.

• Redis streams let you read data as a named consumer,
• check whether that consumer processed the data,
• and reassign unprocessed entries to another consumer.

The following shows how to do that.

Create a consumer group: XGROUP

Insert test data

Read with a consumer group: XREADGROUP

Check pending entries: XPENDING

Acknowledge processing: XACK

Check pending entries again

Pending idle time: XINFO CONSUMERS

Claim pending entries for another consumer: XCLAIM

This reassigns only that one entry to another consumer.
After XCLAIM you still need XACK.
Simple XCLAIM syntax:

min-idle-time is in milliseconds.
Then keep reading and processing.

Several consumers on one stream

If one stream has a lot of work and processing is slow, you can assign several consumers as below.
For example, if 3 entries (Data-1,2,3) arrive per second and each takes 1 second to process, use 3 consumers (Consumer-A,B,C) in separate applications.