Regexp tokenizers are on our TODO list (no concrete delivery date exists for this yet).

@rschu1ze Nice to hear it is in TODO list.

One alternative we are seeing is having additional field for the tokens and create inverted index for the additional field as follows,

CREATE TABLE webapp_logs_final
(
    `event_time` DateTime,
    `raw_message` String,
    `search_tokens` String MATERIALIZED lower(arrayStringConcat(extractAll(raw_message, '([A-Z][a-z]+|[A-Z]+|[a-z]+|[0-9]+)'), ' ')),
    INDEX inv_msg_idx search_tokens TYPE text(tokenizer = 'splitByNonAlpha') GRANULARITY 1
)
ENGINE = MergeTree
ORDER BY event_time

Is it possible to support " lower(arrayStringConcat(extractAll(raw_message, '([A-Z][a-z]+|[A-Z]+|[a-z]+|[0-9]+)') " in the tokenizer preprocessor section so that the additional field can be avoided till it properly implemented. The mentioned regex will generate tokens for alphabets - numbers split and camel case words. Since the values are kept in array can be used in Phrase queries too.

doc1 - rawmessage = "UserID123 Failed due to login problem"
doc2 - rawmessage = "UserID123 LoginFailed"

SELECT * FROM webapp_logs_final WHERE hasToken(search_tokens, '123') AND hasToken(search_tokens, 'failed') AND (search_tokens LIKE '%123 login%'). 

Since the hasToken queries filters the documents only relevent documents are queries for LIKE operator. - Does it make sense?

Instead of discussing in a nested sub-thread of a question that has been asked and answered one year ago, could you please open a proper feature request or bug report with a minimal reproducible example?